The Challenge

This year the Challenge will run on D3, a multimodal dataset that contains 9.2M generated images, generated with four SoTA diffusion model generators. Each image is generated starting from a LAION-400M caption, thus referring to a realistic textual description.

The dataset

Existing deepfake detection datasets are limited in their diversity of generators and quantity of images. For this year’s challenge, we release a new dataset that can support learning deepfake detection methods from scratch. Our Diffusion-generated Deepfake Detection dataset (D3) contains nearly 2.3M records and 11.5M images. Each record in the dataset consists of a prompt, a real image, and four images generated with as many generators. Prompts and corresponding real images are taken from LAION-400M, while fake images are generated, starting from the same prompt, using different text-to-image generators.

For the training and validation set, we employ four state-of-the-art opensource diffusion models, namely Stable Diffusion 1.4 (SD-1.4), Stable Diffusion 2.1 (SD-2.1), Stable Diffusion XL (SD-XL), and DeepFloyd IF (DF-IF). While the first three generators are variants of the Stable Diffusion approach, DeepFloyd IF is strongly inspired by Imagen and thus represents a different generation technique. The test set, contains the same generators plus a collection of SoTA generators not included in the training set, also including GAN-based generators. Additionally it is organized in two spilts, with and without transformations. The submission of the prediction of both splits is required. The only task of the challenge will be deepfake detection, i.e. distinguishing real from generated images.

With the aim of increasing the variance of the dataset, images have been generated with different aspect ratios, 2562, 5122, 640x480, and 640x360. Moreover, to mimic the distribution of real images, we also employ a variety of encoding and compression methods (BMP, GIF, JPEG, TIFF, PNG). In particular, we closely follow the distribution of encoding methods of LAION itself, therefore favoring the presence of JPEG-encoded images.

Data download

Training, validation splits, and the test set are already available as an Huggingface Dataset.

🤗 Training and Validation splits

🤗 Test set

The Submission Website, together with the public leaderboard, is now available at Submission. You can find more information about submission format, training/testing data, rules of the challenge in the ELSA Benchmark platform. This challenge is desribed in the ELSA website as “Track 2 - Classification of totally generated images on multiple generators”.

FAQ

  • What does the test set comprise?: The test set is composed of IDs and images, both authentic and fake. Participants can download two splits of the set: one with original images and another with the images altered (transformed with augmentation techniques). The evaluation of both splits is required for the evaluation.
  • How should submissions be formatted?: Submissions must be in the form of a JSON file. For guidance, you can refer to a sample JSON file at Random Json File.
  • Can I train my model on other dataset (different from D3)?: Participants are required to train their models exclusively on the provided training dataset. The evaluation must be conducted on the provided test datasets. The use of pre-trained models is permitted, as long as the pre-training originates from a distinct domain of deepfake detection (for example pre-trained ResNet on ImageNet, CLIP etc…). For this reason if required, participants must provide all the necessary code for training, and pre-set random seeds to ensure accurate reproducibility. All submitted models and results are subject to re-evaluation.

Timeline

  • Release of Training and validation splits: Jan 21st, 2024
  • Submission site opens: March 20th, 2024
  • Challenge closes: June 1st, 2024
  • Workshop date: June 17-18, 2024

Contacts

For any need or help, you can send a message to elsa_media_analytics@googlegroups.com. The challenge organizers will reply as soon as possible.

Organizers

This challenge is organized as part of the ELSA EU project, in collaboration with Leonardo SpA.