Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
|
education [2021/04/06 16:05] fablpd |
education [2021/09/08 10:00] fablpd |
||
|---|---|---|---|
| Line 8: | Line 8: | ||
| \\ | \\ | ||
| - | * [[education/ca_2020|Concurrent Algorithms]] (theory & practice) | + | * [[education/ca_2021|Concurrent Algorithms]] (theory & practice) |
| * [[education/da|Distributed Algorithms]] (theory & practice) | * [[education/da|Distributed Algorithms]] (theory & practice) | ||
| \\ | \\ | ||
| Line 47: | Line 47: | ||
| * **Byzantine-resilient heterogeneous GANs**: Byzantine-resilient federated learning has emerged as a major theme over the last couple of years, in grand part due to the need to distribute machine learning across many nodes, due to performance and privacy concerns. Until now it has focused on training a single model across many workers and many parameter serves. While this approach has brought on formidable results - including in GAN training, the topic of efficient, distributed and byzantine-resilient training of heterogeneous architectures remain relatively unexplored. In the context of Generative adversarial networks (GANs), such learning is critical to training light discriminators that can specialize in detecting specific featuers of generator-generated images. The goal of this project will be to investigate the potential for GAN training process poisonning by malicious discriminators and generators and investigate efficient protocols to ensure the training process robustness. You will need to have experience with scientific computing in Python, ideally with PyTorch experience, and notions of distributed computing. Contact [[https://people.epfl.ch/andrei.kucharavy|Andrei Kucharavy]] for more information. | * **Byzantine-resilient heterogeneous GANs**: Byzantine-resilient federated learning has emerged as a major theme over the last couple of years, in grand part due to the need to distribute machine learning across many nodes, due to performance and privacy concerns. Until now it has focused on training a single model across many workers and many parameter serves. While this approach has brought on formidable results - including in GAN training, the topic of efficient, distributed and byzantine-resilient training of heterogeneous architectures remain relatively unexplored. In the context of Generative adversarial networks (GANs), such learning is critical to training light discriminators that can specialize in detecting specific featuers of generator-generated images. The goal of this project will be to investigate the potential for GAN training process poisonning by malicious discriminators and generators and investigate efficient protocols to ensure the training process robustness. You will need to have experience with scientific computing in Python, ideally with PyTorch experience, and notions of distributed computing. Contact [[https://people.epfl.ch/andrei.kucharavy|Andrei Kucharavy]] for more information. | ||
| + | \\ | ||
| + | |||
| + | * **GANs with Transformers**: Since their introduction in 2017, the Transformer architecture revolutionized the NLP machine learning models. Thanks to the | ||
| + | scalability of self-attention only architectures, the models can now | ||
| + | scale into trillions of parameters, allowing human-like capacities of | ||
| + | text generation. However, they are not without their own shortcomings, | ||
| + | notably due to their max-likelihood training mode over data that | ||
| + | contains potentially undesirable statistical associations. | ||
| + | An alternative approach to generative learning - Generative Adversarial | ||
| + | Networks (GANs) - perform remarkably well when it comes to images, but | ||
| + | have until recently struggled with texts, due to their sequential and | ||
| + | discrete nature that is not compatible with gradient back-propagation | ||
| + | they need to train. Some of those issues have been solved, but a major | ||
| + | one - their scalability due to usage of RNNs instead of pure | ||
| + | self-attention architectures. | ||
| + | Previously, we were able to show that it is impossible to trivially | ||
| + | replace RNN layers with Transformer layers | ||
| + | (https://arxiv.org/abs/2108.12275, presented in RANLP2021). This project | ||
| + | will be building on those results and attempting to create stable | ||
| + | Transformer-based Text GANs based on the tricks known to stabilize | ||
| + | Transformer training or to attempt to theoretically demonstrate the | ||
| + | inherent instability of Transformer-derived architectures in adversarial | ||
| + | regime. | ||
| + | |||
| + | You will need a solid background knowledge of linear algebra, | ||
| + | acquaintance with the theory of machine learning, specifically neural | ||
| + | networks, as well as experience with scientific computing in Python, | ||
| + | ideally with PyTorch experience. Experience with NLP desirable, but not | ||
| + | required. | ||
| + | |||
| \\ | \\ | ||