There is a growing demand for AI systems that justify their decisions. In my talk, I will present recent results on adding explainability to black box algorithms as well as ways to design transparent AI methods.  We would discuss the implications of such methods in science and technology and how explainability can provide a feedback mechanism that enhances the performance of AI methods.

AI research keeps accelerating, with dozens of thousands of papers published each year. How does this relate to applying AI in the industry? The talk will start with a brief overview of key recent AI techniques that could apply to a wide variety of real-life problems. It will then provide examples for utilizing such new approaches to develop innovative AI-based products internally at Intel. Intel’s internal AI group (IT AI) transforms the company’s critical operations using AI, from processor architecture and design, through manufacturing to sales. This affects almost every processor that Intel provides, yielding both a 9-digit value for the company and better products for its customers. The talk will discuss video, text and tabular data use-cases, which increase the products quality and the operational efficiency. One of these applications was developed in collaboration with the Stanford AI lab.

Supervised NLP tasks are often addressed today in an end-to-end manner, where a model is trained (or fine tuned) on input-output instances of the full task. However, such an approach may exhibit limitations when applied to fairly complex tasks. In this talk, I will discuss an alternative approach where a complex task is decomposed to its inherent subtasks, each addressed by a targeted model, and will illustrate it for the challenging application of multi-document summarization (MDS). Notably, the decomposition approach becomes particularly appealing when targeted training data for modeling specific subtasks can be derived automatically from the originally available “end-to-end” training data for the full task, as I will show for the MDS case. Additionally, I will describe a separate contribution, namely a targeted Cross-Document Language Model (CDLM). This model is pre-trained specifically to model cross-document relationships supporting diverse tasks in this setting, and is also leveraged within our decomposed MDS architecture.

The field of NLP has undergone a revolution in the past few years geared by the use of very large language models (LMs) that can learn to perform language understanding tasks given only a few examples. In this talk, I will describe two recent projects that use neural retrievers to address some shortcomings of large LMs. First, while large LMs do well given a short piece of text, it is difficult to capitalize on their advantages for tasks that are at the corpus level (say all of Wikipedia). I will describe a self-supervised method for retrieving paragraphs from a large corpus, such as Wikipedia, that performs well even without any training examples. Second, the behavior of very large LMs strongly depends on the examples that are given to them as input (this is often termed in-context learning). We present a method for retrieving examples from the training set that maximizes the performance of the LM on a downstream language understanding task. Our approaches make it easier to build new question answering models for arbitrary corpora and to interact with large LMs that are provided as a service by commercial companies.

The typical approach in natural language processing is to use one-size-fits-all representations, obtained from training one model on very large text collections. While this approach is effective for those people whose language style is well represented in the data, it fails to account for variations between people, and thus may lead to worse performance for those in the minority. In this talk, I will challenge the one-size-fits-all assumption, and show that (1) we can identify words that are used in significantly different ways by speakers from different cultures; and (2) we can effectively use information about the people behind the words to build better natural language processing models.

Have you ever wondered how an autonomous bulldozer chooses its next actions to optimally preform the grading task?
In my talk I will share our methodology and how we use reinforcement learning, behavior cloning, contrastive learning and other techniques to train a policy for autonomous bulldozers. 

In addition, I will show our simulation and prototype robot which we used to test and validate our algorithm. 
 

In this talk we demonstrate how attackers can apply split-second phantom adversarial attacks against AI of advance driving assistance systems, causing two commercial advanced driving-assistance systems (Tesla Model X and Mobileye 630) to trigger a sudden stop in the middle of the road, apply the brakes, and issue false notifications. A countermeasure consisting of four neural networks will also be presented that assesses the authenticity of a detected object.

Existing analyses of optimization in deep learning are either continuous, focusing on variants of gradient flow (GF), or discrete, directly treating variants of gradient descent (GD).  GF is amenable to theoretical analysis, but is stylized and disregards computational efficiency. The extent to which it represents GD is an open question in deep learning theory.  My talk will present a recent study of this question.  Viewing GD as an approximate numerical solution to the initial value problem of GF, I will show that the degree of approximation depends on the curvature around the GF trajectory, and that over deep neural networks (NNs) with homogeneous activations, GF trajectories enjoy favorable curvature, suggesting they are well approximated by GD. I will then use this finding to translate an analysis of GF over deep linear NNs into a guarantee that GD efficiently converges to global minimum *almost surely* under random initialization. Finally, I will present experiments suggesting that over simple deep NNs, GD with conventional step size is indeed close to GF.  An underlying theme of the talk will be the possibility of GF (or modifications thereof) to unravel mysteries behind deep learning.

The talk is based on a paper recently published as spotlight in NeurIPS 2021 (joint work with Omer Elkabetz).
 

The theory of deep learning focuses almost exclusively on supervised learning, non-convex optimization using stochastic gradient descent, and overparametrized networks. It is common belief that the optimizer dynamics, network architecture, initialization procedure, and other factors tie together and are all components of its success. 
I'll describe our recent work relating classical online learning theory to deep learning in which we decouple optimization, regret/generalization and expressiveness. We give agnostic and online learning guarantees for fully-connected deep neural networks with nonlinear activations. We quantify convergence and regret guarantees for any range of parameters and allow any optimization procedure, such as adaptive gradient methods and second order methods. 
As an application, we derive provable algorithms for deep control and reinforcement learning in the online and episodic settings. 
Based on joint work with Xinyi Chen, Edgar Minasyan and Jason Lee (edited) 

One of the great challenges of modern AI theory is to explain the success of overparameterized systems that learn to generalize even when optimizing over far more free parameters than examples. This success is often attributed to algorithmic traits such as inductive bias, implicit/explicit regularization, linear stability and more. 
A great test case, to study how such algorithmic traits allow learning, is the classical setting of stochastic convex optimization. Indeed, classical results already demonstrate how an algorithm can learn, even in the overparameterized regime, as long as the population loss is convex. Surprisingly, this is possible even when there doesn't seem to be any effective bound on the number of parameters in the model, or the Rademacher complexity of the class.  But, is that due to some type of implicit regularization? Flatness of the minima? Or even stability? What allows learning algorithms to succeed in the convex case? 
In this talk I will describe our recent analyses for the generalization of algorithms such as stochastic gradient descent, gradient descent, and, generally, first order methods. Through these works we can shed light on potential techniques to prove generalization in overparameterized settings, and revisit notions such as capacity, stability and regularization as well as their role in generalization.

Based on joint works with Idan Amir, Assaf Dauber, Meir Feder, Tomer Koren, Yishay Mansour, and Uri Sherman. 
 

In the past decade, deep learning has completely revolutionized AI.  In this talk, I will explain what deep learning is, why it works, what it has done for us so far, and what it is likely to do in the future.

Neural networks (NN) are often designed for pointwise predictions, but can also be designed to predict distributions. There are NN architectures specifically designed for distribution prediction (such as mixture-density networks), but we might have an architecture we designed for pointwise prediction, when we decide we need distribution prediction as well. In Meta, for example, we use NNs to predict the number of views content will get, but for reviewing potentially-harmful content, it is the upper-quantiles of the view distribution that turn out to be relevant.
 
In this short talk we will cover two "free lunch" methods for adapting existing NNs for distribution prediction: variational inference and MC dropout (the latter more commonly used for regularizing models). We will also cover two types of uncertainty, aleatoric and epistemic, roughly corresponding to randomness in the way the data was generated, and randomness in the way we algorithmically use the data. Finally, we will discuss the relationships between the two methods and the two types of uncertainty.
 

There is an increasing demand within consumer-neuroscience (or neuromarketing) for objective neural measures to quantify consumers’ preferences and predict responses to marketing campaigns. However, the properties of EEG raise difficulties for preference prediction: small datasets, high dimensionality, elaborate manual feature extraction, intrinsic noise, and between-subject variations. We aimed to overcome these limitations by combining unique techniques within a Deep Learning (DL) framework, while providing interpretable results for neuroscientific and decision-making insight. In this study, we developed a DL model to predict subject-specific preferences based on their EEG data. In each trial, 213 subjects observed a product’s image, from 72 possible products, and then reported how much they were willing to pay (WTP) for the product. The DL employed EEG recordings from product observation to predict the corresponding reported WTP values. Our results showed 75.09% test accuracy in predicting high vs. low WTP, surpassing other models and a manual feature extraction approach. Meanwhile, network visualizations provided the predictive frequencies of neural activity, their scalp distributions, and critical timepoints, shedding light on the neural mechanisms involved with evaluation. In conclusion, we show that DLNs may be the superior method to perform EEG-based predictions, to the benefit of decision-making researchers and marketing practitioners alike.

IP hijack attacks are strong attacks on the Internet routing system that allows an attacker to deflect traffic through its network and form man-in-the-middle attack.  As such these attacks are often undetected, which allows the attacker, e.g., to perform long term espionage.
To detect such attacks we suggest several tools that are based on deep learning, and enable to build robust detection tools including: supervised DL based on network level (such as BGP announcements) supervised and  non-supervised methods and a method, which is based , for the first time, on route geography.
 

5G Network Data Analytics Function (NWDAF)  provides the ability to collect variety of data from all over the mobile 5G Network .
On top of this data specific network data analytics are generated for insights and actions .
In this talk we are going to go through NWDAF NF Load prediction use case to show how we tradeoff between latency and accuracy.
Will show practical tips and tricks to improve latency and our ML tradeoff selector.
 

In this talk I will share our journey at Shops on creating product embeddings, I will discuss tradeoffs of using internal Meta systems and services vs. customized ones. I will discuss how different products (Instagram/Facebook) affect our model selection and optimization methods, and how all of these can be taken into consideration to provide our customers with the best experience.

The quest for algorithms that enable cognitive abilities is an integral part of machine learning and appears in many facets, such as virtual assistants and visual reasoning. A cognitive system must be capable of processing details from the multiple sensors that pound a device's computation engine. Interpreting observed objects requires an understanding of their semantics. Additionally, it must be sensitive to and pick out relevant nuances relevant to the task. During this presentation, we will present ways to assess and improve perception. We will explore ways to leverage large-scale models (such as CLIP). As a final step, we propose a novel attention mechanism, called Factor Graph Attention, which can operate on any data utility and distinguish useful signals from distracting ones. Our discussion will focus on the limitations of current methods: (i) models may solve the dataset, but not the task directly; (ii) supervised methods are limited by the curated datasets. Further, we demonstrate novel arithmetic capabilities to reason over visual data, as well as state-of-the-art performance on various tasks, such as visual dialog.

It is said that “Data is the new Oil”, and indeed in this Age of Data companies and government agencies alike gather data of all types and from numerous sources in an attempt to become data-driven organizations. A large portion of data however originates from a single underlying source: People. Tweets and blog posts are written by humans for humans; purchase transactions and phone call records convey human desires for things and other people; app logs report on how people interact with computers and mobile devices. Data derived from human behaviour is “messy”: it is dynamic, complex and extremely versatile. Humans’ behaviour, as recorded in such digital data channels, changes drastically over time, is influenced by underlying complex social networks, and is conveyed in highly multimodal data streams – posing a significant hurdle to any organization striving to truly base its decisions on operations on its data.
Developed at the MIT Human Dynamics Lab, Social Physics is a novel new scientific approach to data, which uses big data analysis and the mathematical laws of biology to understand the behaviour of human crowds, enabling the development of a fully-automatic platform that can absorb, analyze and merge dynamic data streams of various sources, forms and types. Social Physics is the technological framework behind the Endor analytics engine, serves global financial institutes and government agencies.
 

Absolute camera pose regressors estimate the position and orientation of a camera from the captured image alone. As such, they offer a fast, lightweight and standalone alternative to localization pipelines. Typically, a convolutional backbone with a multi-layer perceptron head is trained using images and pose labels to embed a single reference scene at a time. Recently, this scheme was extended for learning multiple scenes by replacing the MLP head with a set of fully connected layers. In this work, we propose to learn multi-scene absolute camera pose regression with Transformers, where encoders are used to aggregate activation maps with self-attention and decoders transform latent features and scenes encoding into candidate pose predictions. This mechanism allows our model to focus on general features that are informative for localization while embedding multiple scenes in parallel. Our method achieves a new state-of-the-art localization accuracy for pose regression methods across indoor and outdoor benchmarks, surpassing both multi-scene and single-scene absolute pose regressors.

In this talk I will overview our program at Duke University, where we have developed and deployed an app to study behaviors in developmental disorders, autism spectrum disorder in particular, providing scalable tools with state-of-the-art performance thanks to careful co-design of stimuli and ML. The app has been deployed in pediatric clinics and has already collected and analyzed the largest ever dataset of this kind in the field.

The single image super-resolution task is one of the most examined inverse problems in the past decade. In recent years, Deep Neural Networks (DNNs) have shown superior performance over alternative methods when the acquisition process uses a fixed known downsampling kernel-typically a bicubic kernel. However, several recent works have shown that in practical scenarios, where the test data mismatch the training data (e.g. when the downsampling kernel is not the bicubic kernel or is not available at training), the leading DNN methods suffer from a huge performance drop. Inspired by the literature on generalized sampling, in this work we propose a method for improving the performance of DNNs that have been trained with a fixed kernel on observations acquired by other kernels. For a known kernel, we design a closed-form correction filter that modifies the low-resolution image to match one which is obtained by another kernel (e.g. bicubic), and thus improves the results of existing pre-trained DNNs. For an unknown kernel, we extend this idea and propose an algorithm for blind estimation of the required correction filter. We show that our approach outperforms other super-resolution methods, which are designed for general downsampling kernels.

The past two years demonstrated how important data analyses are for policy making directly affecting public health. Following the development and distribution of vaccines, trustworthy and carefully considered analyses have shown to be paramount. Nevertheless, the extent to which causal conclusions can be drawn has been central to the public debate. One key challenge is the ever-changing environment: different variants, vaccine coverage and the pandemic state. These issues characterize a more general target: learning causal effects for infectious diseases.  Within this field, a major challenge is evaluating the impact of prescribing different antibiotics on future antibiotic resistance, yet another global threat. A recent study estimated the number of worldwide deaths attributed to antibiotic resistance in 2019 to be 5 million.  
In this talk, I will present these challenges and discuss our approach towards overcoming them in several projects concerning antibiotic effects on resistance and vaccine effects.
 

Our blood carries multiple molecules that can represent an individual's condition in health and disease. Using advanced sequencing technologies, we monitor tiny amounts of DNA in blood in order to observe the early stages of cancer and examine embryo development. Our deep learning algorithms applied on billions of DNA molecules lead to the classification of cancer types, early detection of colorectal cancer, and identification of single-point mutations during the first trimester of embryonic development. Our work will assist clinical teams in assessing the patient’s medical illness and to devise precision therapy based on deep evaluation of an individual’s molecular status.

The human genome consists of three billion characters of A,T,G, and C bases, and two copies that we inherit from each parent establish the instructions for our biochemistry. Analyzing genome sequence data is important to resolving cases of genetic disease, which affects roughly 6% of individuals, and understanding broad predisposition to common diseases. In this talk, we will discuss deep learning approaches to analyze genomic data. This includes DeepVariant, a widely-used, open-source method for variant detection using Convolutional Neural Networks, DeepConsensus, an open-source method for sequence error correction using Transformers, as well as DeepNull and ML-based phenotyping which combine multi-layer perceptrons and Convolutional Neural Networks with traditional statistical approaches to discover new genetic associations. The talk will focus on how the deep learning approaches were adapted to the biological data, and will briefly touch on results of external application of the methods.

The rapid development of artificial intelligence tools has led to unprecedented impact on almost any field of life, and specifically on image-to-image translation tasks. However, these techniques typically require large amounts of data which may not be available in fields such as medical imaging. A promising alternative is the mixture of signal models with powerful denoisers, providing a general solution for imaging tasks which can operate in low-data regimes. Two frameworks which follow this approach are regularization-by-denoising (RED) and plug-and-play priors (PnP) that have shown state-of-the-art performance in various imaging problems. However, the stability of common RED and PnP methods, which is crucial in medical applications, cannot be completely guaranteed. Here we introduce potential-driven neural networks where the image-to-image translation maps are built as gradients of neural networks. Specifically, we train ponterial-driven denoisers and employ them to solve model-based imaging problems via a convergent gradient-descent scheme. The resultant technique provides comparable performance to RED and PnP methods with provable stability.   

AI algorithms for personalized medicine necessitate large DNA databases of patient information.  Here, I will show that genetic data is intrinsically and easily identifiable using simple internet searches in consumer genetic websites. These websites offer powerful parametric models that predict genealogical relationships using the genetic data and allow finding distant relatives of the target, which can eventually lead to target identification. By inspecting genetic data of 1.28 million individuals tested with consumer genomics, we investigated the power of this technique. We project that about 60% of the searches for individuals of European descent will result in a third-cousin or closer match, which theoretically allows their identification using demographic identifiers. Moreover, the technique could implicate nearly any U.S. individual of European descent in the near future. We demonstrate that the technique can also identify research participants of a public sequencing project. On the basis of these results, we propose a potential mitigation strategy and policy implications for human subject research.

Deep neural networks provide unprecedented performance gains in many real-world problems in signal and image processing. Despite these gains, the future development and practical deployment of deep networks are hindered by their black-box nature, i.e., a lack of interpretability and the need for very large training sets. On the other hand, signal processing has traditionally relied on classical statistical modeling techniques that utilize mathematical formulations representing the underlying physics, prior information and additional domain knowledge. Simple classical models are useful but sensitive to inaccuracies and may lead to poor performance when real systems display complex or dynamic behavior. Here we introduce various approaches to model based learning which merge parametric models with optimization tools leading to efficient, interpretable networks from reasonably sized training sets. We will consider examples of such model-based deep networks to image deblurring, image separation, super resolution in ultrasound and microscopy, and finally we will see how model-based methods can also be used for efficient diagnosis of COVID19 using X-ray and ultrasound.

Given the current population of Covid mutations, what is the probability that we encounter a new variant that has not been seen before? This fundamental question is known as the missing mass problem. In this work we introduce a new scheme for missing mass estimation. Our proposed framework provides novel risk bounds and improves upon currently known methods. Importantly, it is easy to apply and does not require additional modeling assumptions. This makes it a favorable choice for many practical applications.

The sequence alignment problem is one of the most fundamental issues in bioinformatics and indeed a plethora of methods has been devised to tackle it. In practice, these methods are applied with the same parameters (i.e., default configuration) on every given input, implicitly assuming a single evolutionary model. This is clearly an oversimplification of biological reality.
Here we introduce AlignNLP, a state-of-the-art methodology for aligning sequences using a deep neural network. AlignNLP accounts for the possible variability of the evolutionary process among different lineages by using an ensemble of transformers. Each transformer is trained on millions of samples from different evolutionary models, leading to a superb alignment accuracy, overperforming commonly used methods, such as MAFFT and Prank. 

Pulmonary embolism (PE) is a common life-threatening condition with a challenging diagnosis, as patients often present with nonspecific symptoms. Prompt and accurate detection of PE and specifically an assessment of its severity are critical for managing patient treatment. We introduce diverse multimodal fusion models that are capable of utilizing weakly-labeled multi-modal data, combining both volumetric pixel data and clinical patient data for automatic risk stratification of PE. The best performing multimodality model achieves an AUC of 0.96 for assessing PE severity, with a sensitivity of 90% and specificity of 94%. To the best of our knowledge, this is the first study that attempted to automatically assess PE severity.

The aim of this contribution is a "regulation and governance" reading of the new EU initiative on regulating Artificial Intelligence. Thus its focus is not on the "first order" fundamental rights perspective but rather the "second order" mechanisms that are necessary for their effective application on the ground. 

Various efforts for early detection and prompt isolation of the COVID-19 pandemic are pivotal to breaking transmission chains and containing outbreaks. In this talk, I will briefly discuss our four projects that utilize data of a large-scale for early detection of 1) hotspots and regional outbreaks using mobility data from cellphone devices, 2) early signs of infections using wearable devices, 3) positive test outcomes using historical electronic medical records, and 4) deteriorations of hospitalized patients using rich medical data.  Then, I will elaborate on one of the projects and its potential applicability for policy determination.

COVID19 pandemic challenged society and organizations to make a trade-off between individual health safety on one hand, and socio-economic progress on the other. In a way, we're all grappling to return to a new normal. But, what would that new normal be? How and when can we return to that new normal without compromising our health safety? Uncertainties related to new variants, vaccines and their efficacies, loss of immunity, and socio-economic factors make such questions difficult to answer. We developed a novel configurable digital twin that can be contextualized for a city or an organization to predict possible disruptions of business-as-usual activities due to the COVID pandemic. It also helps to explore various what-if scenarios for defining strategies towards a safer return to new normalcy. Our approach combines multiple types of system modelling & simulation techniques and adopts established data science concepts. We demonstrated the efficacy of our approach in the context of an Indian city and an organization. In this talk, I will present our approach, early experiences, and learnings.  

Artificial intelligence techniques such as machine and deep learning have large potential to better exploit the rich information that can be derived from health and healthcare data to promote public and individual health. It can change our healthcare system from a reactive to a proactive one (towards prevention), and by learning from previous patients we can personalize treatment, leading to "precision medicine". 

As the quality of many AI techniques crucially depends on the data on which it is trained, an important prerequisite for the successful implementation of AI in healthcare is access to high quality data. This is complex in the health domain, where there is fragmentation of expertise, data and resources. Therefore, in the Dutch health and life sciences domain different stakeholders have established a joint vision towards a Dutch health data infrastructure, and launched the Health RI initiative (www.health-ri.org). Health RI pursues the integration of highly diverse collections of longitudinal health and biomedical data, e.g. generated at different hospitals and research centers, to empower researchers and organizations to develop better personalized medicine and health solutions. The covid pandemic has shown the importance of such an approach, and in this presentation the steps towards a national Covid data portal will be presented.
 

In this talk I will cover how to get a GAN to run on a mobile device and give a consumer control over its results. At Lightricks we build tools for the creative process. These tools need to be consistent, controllable and accessible in order to succeed in the market. GANs are amazing machines with many capabilities, but turning these wild horses into tools used by millions on mobile devices is a challenge. I will describe the stages of turning GANs into product grade features that can even work on the edge. The hurdles of better data, model efficiency and the tradeoffs between control and expression when choosing your architecture.

While image editing and manipulation tools have seen remarkable progress, video editing remains a difficult task that poses two key challenges: (i) edits need to be applied in a temporally consistent manner to all frames, and (ii) editing interfaces need to be able to represent temporal content in an intuitive manner.  Thus, video editing has been largely restricted to the domain of professionals. In this talk, I’ll present a new method that tackles these challenges, and allows easy and intuitive editing of everyday videos by novice users. 
The pillar of the approach is a novel decomposition of the input video into a set of layered 2D atlases ('texture maps'), each providing a unified representation of an object/background over the entire video. Using the learned decomposition, we can simply edit the 2D atlases (or a single frame), and automatically propagate the edits to the entire video. By operating purely in 2D, our method does not require any prior 3D knowledge about scene geometry or camera poses. I’ll show a variety of exciting editing results including texture mapping, video style transfer, image-to-video texture transfer, and segmentation/labeling propagation.
Project page: https://layered-neural-atlases.github.io/
 

In this talk, we will show how one may use tools from signal processing to improve neural network performance in various applications including image super-resolution, robustness to label noise, better representation capabilities and more.