Rishabh Jain
I am a rising second year Computer Science MS (specialization in Machine Learning) student at Georgia Tech advised by Prof. Dhruv Batra and work closely with Prof. Devi Parikh. I work at the intersection of software engineering and AI research, solving problems which lie at the intersection of Machine Learning, Computer Vision and Natural Language Processing.
I am excited about building tools for reproducible AI research and scalable software which will help in training, visualizing and evaluating machine learning models in real-time on static and dynamic datasets.
I am currently working as a Graduate Research Assistant with Prof. Dhruv Batra and Prof. Devi Parikh for simplifying and standardizing the process of evaluating Reinforcement Learning (RL) models by developing and managing a widely adopted evaluation platform, EvalAI, for reproducibility of results, maintaining evaluation consistency, evaluating of model’s code instead of predictions from the model, and to measure constant progress on pushing the frontiers of AI.
I also lead an open source organization, CloudCV, where we are building several open-source softwares for reproducible AI research. Previously, I spent a year as a visiting research scholar in Machine Learning and Perception Lab at Georgia Tech.
I am graduating in May 2021 and am on the job market. Please feel free to reach out to me at rishabhjain@gatech.edu.

News


Projects

EvalAI: Towards Better Evaluation Systems for AI Agents
Star   Fork   Code
Built an open source platform for evaluating and benchmarking AI models. We have hosted 100+ AI challenges with 10,000+ users, who have created 100,000+ submissions. More than 25 organizations from industry and academia use it for hosting their AI challenges. The project is open source with 120+ contributors, and 2M+ pageviews.. Some of the organizations using it are Google, Facebook, IBM, Intel, eBay, etc. and research labs from Stanford, CMU, MIT, Georgia Tech, etc. also use it and it's forked version for hosting their internal challenges instead of reinventing the wheel.
VQA: Visual Question Answering Demo
Star   Fork   Code
In Visual Question Answering, given an image and a free-form natural language question about the image (e.g., "What kind of store is this?", "How many people are waiting in the queue?", "Is it safe to cross the street?") the model's task is to automatically produce a concise, accurate, free-form, natural language answer ("bakery", "5", "Yes"). This demo is implemented using Pythia model. It is used by 100K+ users.
Visual Chatbot: A chatbot that can see!
Star   Fork   Code
Built a visual chatbot which can hold a meaningful dialog with humans in natural, conversational language about visual content. Specifically, given an image, a dialog history, and a question about the image, the chatbot will ground the question in image, infer context from history, and answer the question accurately. This demo is implemented using Late Fusion model from CVPR 2017 Paper. It is used by 170K+ users.
12-in-1: Multi-Task Vision and Language Representation Learning Demo
Star   Fork   Code
Built a demo of a single model trained on 12 datasets from four broad categories of tasks including visual question answering, caption-based image retrieval, grounding referring expressions, and multi-modal verification and compared to independently trained single-task models, this represents a reduction from approximately 3 billion parameters to 270 million while simultaneously improving performance by 2.05 points on average across tasks. It is used by 20K+ users.
Trick or TReAT: Thematic Reinforcement for Artistic Typography Demo
Given an input word (e.g.exam) and a theme (e.g.education), the individual letters of the input word are replaced by cliparts relevant to the theme which visually resemble the letters - adding creative context to the potentially boring input word.
JSS InfoConnect: Information Center for College
Star   Fork   Code
Web based officially recognized application that acts as a medium for interaction between all students, faculties and management of JSSATE, Noida. It serves 10K+ requests per day, 20K+ registered users, 20K+ notices/documents were uploaded since it’s launch in 2016.

Publications

Dialog without Dialog: Learning Image-Discriminative Dialog Policies from Single-Shot Question Answering Data
Michael Cogswell*, Jiasen Lu*, Rishabh Jain, Stefan Lee, Dhruv Batra, Devi Parikh
EvalAI: Towards Better Evaluation Systems for AI Agents
Deshraj Yadav, Rishabh Jain, Harsh Agrawal, Prithvijit Chattopadhyay, Taranjeet Singh, Akash Jain, Shiv Baran Singh, Stefan Lee, Dhruv Batra
nocaps: novel object captioning at scale
Harsh Agrawal*, Karan Desai*, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, Peter Anderson
Evaluating visual and text explanations in an interactive, goal-driven human-AI task
Arjun Chandrasekaran, Rishabh Jain, Karan Desai, Kerry Moffitt, Jeff Miller, David Diller, Bill Ferguson, Devi Parikh
(* denotes equal contribution)

Experience

Team Lead  
(Jan 2019  -  Present)
Leading a team of 15+ contributors to actively maintain CloudCV Project which aims to make AI research more reproducible.
Graduate Research Assistant, Machine Learning and Perception Lab
Supervised by: Prof. Dhruv Batra and Prof. Devi Parikh  
(Aug 2019  -  Present)
EvalAI: Built an open source platform called as EvalAI for evaluating and benchmarking AI models. We have hosted 100+ challenges with 10,000+ users, who have created 100,000+ submissions. The project is open source with 120+ contributors, 1k+ stars, 500+ forks. More than 25 organizations are using it including Google, Facebook, IBM, Intel, eBay etc. and research labs from MIT, Stanford, CMU, Georgia Tech, etc. also use it and it's forked version for hosting their internal challenges instead of reinventing the wheel.
GuessWhich: Evaluating the role of interpretable explanations towards making a model predictable to a human. We studied if the textual or visual explanations from an AI model in the context of an interactive, goal-driven, collabora-tive human-AI task help humans to predict it’s behavior.
Software Engineering Intern, Structured Data and Applied Research Team
Supervised by: Dr. Roman Maslovskis and Dr. Uwe Mayer  
(May 2020  -  Aug 2020)
Evaluating and Predicting attribute values in listings: Given an image, and a text description about the listing on eBay, the task is to predict the missing attributes in the listing. For instance, predicting the missing color or brand attribute in the listing. I built an end-to-end system for processing visual and textual data along with training AI models. I also trained an early fusion model of image and text data which outperformed the uni-modal models of image and text by 5% and 22% on the test dataset for color attribute and by 20% and 4% on the test dataset for brand attribute.
Visiting Research Scholar, Machine Learning and Perception Lab
Supervised by: Prof. Dhruv Batra and Prof. Devi Parikh  
(Aug 2018  -  June 2019)
To encourage the development of image captioning models that can learn visual concepts from alternative data sources, such as object detection datasets, we present the first large-scale benchmark for this task. Dubbed nocaps, for novel object captioning at scale, our benchmark consists of 166,100 human-generated captions describing 15,100 images from the Open Images validation and test sets containing more than 500 objects, out of which more than 400 objects are never described in COCO captions dataset.
Google Summer Of Code (GSoC)
Organization Administrator (2020, 2019), Organization Mentor (2018), Student Developer (2017) 
(May 2017   -  Aug 2020)
2017: I was selected as GSoC student where I developed new features for hosting AI challenges in streamlined manner, implemented REST-API’s, frontend and several analytics features for both participants and hosts in EvalAI.
2018: Mentored a student to design a command line tool (EvalAI-CLI) for EvalAI which lets the participants to install and use EvalAI as a python package.
2019: Led a team of 10+ mentors and students under the CloudCV organization to setup the pipeline for evaluation of code upload challenges, leading the development of new UI and automating the existing submission worker using AWS Fargate on EvalAI.
2020: Leading a team of 10+ mentors and students under the CloudCV organization to automate the pipeline for evaluation of code upload challenges, release new UI and, streamlining AI challenge creation on EvalAI.
Google Code In (GCI)
Organization Administrator (2019, 2018), Organization Mentor (2017) 
(Nov 2017   -  Jan 2020)
2017: Applied with CloudCV as a mentoring organization and got it accepted to mentor for the first time in Google Code-In. I mentored high school students on open-source projects in frontend, backend and DevOps.
2018: Led a team of 10+ mentors to mentor high school students on open-source projects EvalAI, Fabrik and, Orgami.
2019: Led a team of 10+ mentors to mentor high school students on open source projects EvalAI, EvalAI-CLI, EvalAI-ngx.
Python Software Society of India (PSSI)
Open Source Contributor  
(July 2016  -  Oct 2016)
Implemented several features in the Junction project including the feature of creating and managing users profile. Junction is an open-source software to manage proposals, reviews, schedules, feedback during the PyCon conference.
iAugmentor Labs
Software Developer Intern, Machine Learning  
(Jun 2016  -  Aug 2016)
Developed a classifier using SVM to detect and classify videos containing an answer of a job interview question as an input and outputs a confidence score on the smiling behaviour of a person in the video.
Nibble Computer Society (NCS)
Organizing Member  
(Feb 2015  -  May 2018)
Organized multiple code labs, seminars, workshops on OOPs, Advanced C, C++, Google Summer of Code, Git etc.and mentored 30+ undergraduate students on software development. I was also responsible for organizing college’sannual techno-cultural fest Zealicon.

Invited Talks


Education

Georgia Institute of Technology
Masters in Computer Science
Georgia Institute of Technology, Atlanta, USA  
(Aug 2019  -  Present)
  • Specialization in Machine Learning
  • Current GPA: 4.0/4.0
  • Expected Graduation  -  May 2021
Bachelor of Technology in Computer Science and Engineering
JSS Academy of Technical Education, Noida, India  
(Aug 2014  -  May 2018)
  • Passed with an aggregate of 80.4% (with Hons.)
  • Class Rank: 8th out of 150 Students (or top 5%)