Post-Doctoral Research Visit F/M Robust and effective stochastic
Offer DescriptionContexte et atouts du posteThe project will be funded by the PRAIRIE 3IA Institute — ANR-19-P3IA-0001.Travel expenses are covered within the limits of the scale in force.Mission confiéeThe main objective of this project is the understanding and development of robust and effective stochastic optimization methods for training deep learning models.State-of-the-art optimization techniques require careful tuning of several hyperparameters, such as learning rate (schedule), momentum, and weight decay. One aspect of this research project is to investigate and develop adaptive techniques for selecting hyperparameters, in order to reduce the tuning effort. For instance, the stochastic Polyak step size and its recent variants have shown promising results for adaptively setting the learning rate.One challenge in this regard is that theoretically optimal hyperparameter values depend on quantities that are unknown before training. A core part of the project consists of studying online estimation of unknown quantities for hyperparameter selection and exploring its applicability for modern deep learning problem instances.The project also aims to develop a theoretical framework for robust optimization methods that are resilient to outliers and heavy-tailed noise in gradient distributions. This is particularly relevant for transformer models, which have dominated the field in recent years but often suffer from training instabilities. We aim to investigate the interactions between model architectures and data domains in deep learning, focusing on disentangling the effects of transformer models and their input data distribution on gradient outliers.
The recruited person will be co-advised with Adrien Taylor and Francis Bach.Principales activitésMain activities :Conduct theoretical researchRequirementsAdditional InformationWork Location(s)Number of offers available 1 Company/Institute Inria Country France GeofieldWhere to apply WebsiteSTATUS: EXPIRED
France
Thu, 25 Apr 2024 22:15:48 GMT
To help us track our recruitment effort, please indicate in your email/cover letter where (vacanciesineu.com) you saw this job posting.
Location: Italy Salary: Competitive Type: Permanent Main Industry: Search Health, Nursing & Social Services Jobs…
Job title: College Services Advisor Company: Kaplan International Job description College Services AdvisorReference number: JR237405Location:…
Job title: Multiskilled operator Company: Nutreco Job description The Multi-Skilled Operator executes a broad range…
Job title: Postdoctoral Researcher (Ref: Postdoc 2024-16 KV) Company: ICIQ Job description The Institute of…
Job title: Mechanical Supervisor Company: LJB & Co Job description We are assisting a leading…
Location: Chartres - Eure-et-Loir, Centre, France Salary: Competitive Type: Permanent Main Industry: Search Media Jobs…