Agent AI

LLM Agent – Retrieve What You Need: A Mutual Learning Framework for Open-domain Question Answering

Dingmin Wang, Qiuyuan Huang, Matthew Jackson, Jianfeng Gao

Proceedings of Transactions of the Association for Computational Linguistics (TACL) | April 2024, pp. 247-263

Publication Project

Agent AI Towards a Holistic Intelligence

Qiuyuan Huang, Naoki Wake, Bidipta Sarkar, Zane Durante, Ran Gong, Rohan Taori, Yusuke Noda, Demetri Terzopoulos, Noboru Kuno, Ade Famoti, Ashley J. Llorens, John Langford, Hoi Vo, Fei-Fei Li, Katsushi Ikeuchi, Jianfeng Gao

arXiv: 2403.00833 | February 2024

Publication Publication

An Interactive Agent Foundation Model

Zane Durante, Bidipta Sarkar, Ran Gong, Rohan Taori, Yusuke Noda, Paul Tang, Ehsan Adeli, Shrinidhi Kowshika Lakshmikanth, Kevin Schulman, Arnold Milstein, Demetri Terzopoulos, Ade Famoti, Noboru Kuno, Ashley J. Llorens, Hoi Vo, Katsushi Ikeuchi, Fei-Fei Li, Jianfeng Gao, Naoki Wake, Qiuyuan Huang

February 2024

Publication Publication

MindAgent: Emergent Gaming Interaction

Steven Gong, Qiuyuan Huang, Xiaojian Ma, Hoi Vo, Zane Durante, Yusuke Noda, Zilong Zheng, Song-chun Zhu, Demetri Terzopoulos, Feifei Li, Jianfeng Gao

Processings of NAACL 2024 | January 2024

Publication Project

Agent AI: Surveying the Horizons of Multimodal Interaction

Zane Durante, Qiuyuan Huang, Naoki Wake, Ran Gong , Jae Sung Park, Bidipta Sarkar, Rohan Taori, Yusuke Noda, Demetri Terzopoulos, Yejin Choi, Katsushi Ikeuchi, Hoi Vo, Fei-Fei Li, Jianfeng Gao

arXiv: 2401.03568 | January 2024

Publication

Multimodal Agent — Localized Symbolic Knowledge Distillation for Visual Commonsense Models

Jae Sung Park, Jack Hessel, Khyathi Chandu, Paul Pu Liang, Ximing Lu, Peter West, Qiuyuan Huang, Jianfeng Gao, Ali Farhadi, Yejin Choi

Proceedings of NeurIPS 2023 | October 2023

Publication

Logical Transformers: Infusing Logical Structures into Pre-Trained Language Models

Borui Wang, Qiuyuan Huang, Budhaditya Deb, Aaron L. Halfaker, Liqun Shao, Daniel McDuff, Ahmed Awadallah, Dragomir Radev, Jianfeng Gao

Proceedings of ACL 2023 | May 2023

Publication Publication Project

ArK: Augmented Reality with Knowledge Emergent Infrastructure

Qiuyuan Huang, James Park, Abhinav Gupta, Pan Lu, Paul Bennett, Ran Gong, Subhojit Som, Baolin Peng, Owais Khan Mohammed, Chris Pal, Yejin Choi, Jianfeng Gao

Selected as the project in HackBox 2023; Shipped to Office Teams | April 2023

Publication Project

VLC: Training Vision-Language Transformers from Captions

Liangke Gui, Qiuyuan Huang, Alex Hauptmann, Yonatan Bisk, Jianfeng Gao

Proceedings of Transactions on Machine Learning Research | May 2022

Publication Project

KAT: A Knowledge Augmented Transformer for Vision-and-Language

Liangke Gui, Borui Wang, Qiuyuan Huang, Alex Hauptmann, Yonatan Bisk, Jianfeng Gao

NAACL 2022. Long paper, Oral. | January 2022

No.1 of OK-VQA Leaderboard (2021).

Publication

NICE: Neural Image Commenting with Empathy

Kezhen Chen, Qiuyuan Huang, Daniel McDuff, Xiang Gao, Hamid Palangi, Jianfeng Wang, Kenneth Forbus, Jianfeng Gao

EMNLP 2021 | October 2021

Publication

Mapping Natural-language Problems to Formal-language Solutions Using Structured Neural Representations

Kezhen Chen, Qiuyuan Huang, Hamid Palangi, Paul Smolensky, Kenneth D. Forbus, Jianfeng Gao

ICML 2020 | July 2020

Publication Project

Vision-Language Navigation Policy Learning and Adaptation

Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-fang Wang, William Wang, Lei Zhang

IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2020 | February 2020

Publication Publication Project

Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation

Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Weng, William Yang Wang, Lei Zhang

CVPR 2019 | June 2019

Best Student Paper Award in CVPR 2019.

Publication Video Project Project

Hierarchically Structured Reinforcement Learning for Topically Coherent Visual Story Generation

Qiuyuan Huang, Zhe Gan, Asli Celikyilmaz, Oliver Wu, Jianfeng Wang, Xiaodong He

AAAI 2019 | January 2019

Publication Project Project

Turbo Learning for CaptionBot and DrawingBot

Qiuyuan Huang, Pengchuan Zhang, Oliver Wu, Lei Zhang

NeurIPS 2018 | December 2018

PDF Publication Project

LLM Agent – Retrieve What You Need: A Mutual Learning Framework for Open-domain Question Answering

Dingmin Wang, Qiuyuan Huang, Matthew Jackson, Jianfeng Gao

Proceedings of Transactions of the Association for Computational Linguistics (TACL) | April 2024, pp. 247-263

Publication Project

Agent AI Towards a Holistic Intelligence

Qiuyuan Huang, Naoki Wake, Bidipta Sarkar, Zane Durante, Ran Gong, Rohan Taori, Yusuke Noda, Demetri Terzopoulos, Noboru Kuno, Ade Famoti, Ashley J. Llorens, John Langford, Hoi Vo, Fei-Fei Li, Katsushi Ikeuchi, Jianfeng Gao

arXiv: 2403.00833 | February 2024

Publication Publication

An Interactive Agent Foundation Model

Zane Durante, Bidipta Sarkar, Ran Gong, Rohan Taori, Yusuke Noda, Paul Tang, Ehsan Adeli, Shrinidhi Kowshika Lakshmikanth, Kevin Schulman, Arnold Milstein, Demetri Terzopoulos, Ade Famoti, Noboru Kuno, Ashley J. Llorens, Hoi Vo, Katsushi Ikeuchi, Fei-Fei Li, Jianfeng Gao, Naoki Wake, Qiuyuan Huang

February 2024

Publication Publication

MindAgent: Emergent Gaming Interaction

Steven Gong, Qiuyuan Huang, Xiaojian Ma, Hoi Vo, Zane Durante, Yusuke Noda, Zilong Zheng, Song-chun Zhu, Demetri Terzopoulos, Feifei Li, Jianfeng Gao

Processings of NAACL 2024 | January 2024

Publication Project

Agent AI: Surveying the Horizons of Multimodal Interaction

Zane Durante, Qiuyuan Huang, Naoki Wake, Ran Gong , Jae Sung Park, Bidipta Sarkar, Rohan Taori, Yusuke Noda, Demetri Terzopoulos, Yejin Choi, Katsushi Ikeuchi, Hoi Vo, Fei-Fei Li, Jianfeng Gao

arXiv: 2401.03568 | January 2024

Publication

Multimodal Agent — Localized Symbolic Knowledge Distillation for Visual Commonsense Models

Jae Sung Park, Jack Hessel, Khyathi Chandu, Paul Pu Liang, Ximing Lu, Peter West, Qiuyuan Huang, Jianfeng Gao, Ali Farhadi, Yejin Choi

Proceedings of NeurIPS 2023 | October 2023

Publication

Logical Transformers: Infusing Logical Structures into Pre-Trained Language Models

Borui Wang, Qiuyuan Huang, Budhaditya Deb, Aaron L. Halfaker, Liqun Shao, Daniel McDuff, Ahmed Awadallah, Dragomir Radev, Jianfeng Gao

Proceedings of ACL 2023 | May 2023

Publication Publication Project

ArK: Augmented Reality with Knowledge Emergent Infrastructure

Qiuyuan Huang, James Park, Abhinav Gupta, Pan Lu, Paul Bennett, Ran Gong, Subhojit Som, Baolin Peng, Owais Khan Mohammed, Chris Pal, Yejin Choi, Jianfeng Gao

Selected as the project in HackBox 2023; Shipped to Office Teams | April 2023

Publication Project

VLC: Training Vision-Language Transformers from Captions

Liangke Gui, Qiuyuan Huang, Alex Hauptmann, Yonatan Bisk, Jianfeng Gao

Proceedings of Transactions on Machine Learning Research | May 2022

Publication Project

KAT: A Knowledge Augmented Transformer for Vision-and-Language

Liangke Gui, Borui Wang, Qiuyuan Huang, Alex Hauptmann, Yonatan Bisk, Jianfeng Gao

NAACL 2022. Long paper, Oral. | January 2022

No.1 of OK-VQA Leaderboard (2021).

Publication

NICE: Neural Image Commenting with Empathy

Kezhen Chen, Qiuyuan Huang, Daniel McDuff, Xiang Gao, Hamid Palangi, Jianfeng Wang, Kenneth Forbus, Jianfeng Gao

EMNLP 2021 | October 2021

Publication

Mapping Natural-language Problems to Formal-language Solutions Using Structured Neural Representations

Kezhen Chen, Qiuyuan Huang, Hamid Palangi, Paul Smolensky, Kenneth D. Forbus, Jianfeng Gao

ICML 2020 | July 2020

Publication Project

Vision-Language Navigation Policy Learning and Adaptation

Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-fang Wang, William Wang, Lei Zhang

IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2020 | February 2020

Publication Publication Project

Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation

Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Weng, William Yang Wang, Lei Zhang

CVPR 2019 | June 2019

Best Student Paper Award in CVPR 2019.

Publication Video Project Project

Hierarchically Structured Reinforcement Learning for Topically Coherent Visual Story Generation

Qiuyuan Huang, Zhe Gan, Asli Celikyilmaz, Oliver Wu, Jianfeng Wang, Xiaodong He

AAAI 2019 | January 2019

Publication Project Project

Turbo Learning for CaptionBot and DrawingBot

Qiuyuan Huang, Pengchuan Zhang, Oliver Wu, Lei Zhang

NeurIPS 2018 | December 2018

PDF Publication Project

Agent AI Towards a Holistic Intelligence

Qiuyuan Huang, Naoki Wake, Bidipta Sarkar, Zane Durante, Ran Gong, Rohan Taori, Yusuke Noda, Demetri Terzopoulos, Noboru Kuno, Ade Famoti, Ashley J. Llorens, John Langford, Hoi Vo, Fei-Fei Li, Katsushi Ikeuchi, Jianfeng Gao

arXiv: 2403.00833 | February 2024

Publication Publication

Agent AI: Surveying the Horizons of Multimodal Interaction

Zane Durante, Qiuyuan Huang, Naoki Wake, Ran Gong , Jae Sung Park, Bidipta Sarkar, Rohan Taori, Yusuke Noda, Demetri Terzopoulos, Yejin Choi, Katsushi Ikeuchi, Hoi Vo, Fei-Fei Li, Jianfeng Gao

arXiv: 2401.03568 | January 2024

Publication

ArK: Augmented Reality with Knowledge Emergent Infrastructure

Qiuyuan Huang, James Park, Abhinav Gupta, Pan Lu, Paul Bennett, Ran Gong, Subhojit Som, Baolin Peng, Owais Khan Mohammed, Chris Pal, Yejin Choi, Jianfeng Gao

Selected as the project in HackBox 2023; Shipped to Office Teams | April 2023

Publication Project

VLC: Training Vision-Language Transformers from Captions

Liangke Gui, Qiuyuan Huang, Alex Hauptmann, Yonatan Bisk, Jianfeng Gao

Proceedings of Transactions on Machine Learning Research | May 2022

Publication Project

KAT: A Knowledge Augmented Transformer for Vision-and-Language

Liangke Gui, Borui Wang, Qiuyuan Huang, Alex Hauptmann, Yonatan Bisk, Jianfeng Gao

NAACL 2022. Long paper, Oral. | January 2022

No.1 of OK-VQA Leaderboard (2021).

Publication

NICE: Neural Image Commenting with Empathy

Kezhen Chen, Qiuyuan Huang, Daniel McDuff, Xiang Gao, Hamid Palangi, Jianfeng Wang, Kenneth Forbus, Jianfeng Gao

EMNLP 2021 | October 2021

Publication

Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation

Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Weng, William Yang Wang, Lei Zhang

CVPR 2019 | June 2019

Best Student Paper Award in CVPR 2019.

Publication Video Project Project

Hierarchically Structured Reinforcement Learning for Topically Coherent Visual Story Generation

Qiuyuan Huang, Zhe Gan, Asli Celikyilmaz, Oliver Wu, Jianfeng Wang, Xiaodong He

AAAI 2019 | January 2019

Publication Project Project

Turbo Learning for CaptionBot and DrawingBot

Qiuyuan Huang, Pengchuan Zhang, Oliver Wu, Lei Zhang

NeurIPS 2018 | December 2018

PDF Publication Project

LLM Agent – Retrieve What You Need: A Mutual Learning Framework for Open-domain Question Answering

Dingmin Wang, Qiuyuan Huang, Matthew Jackson, Jianfeng Gao

Proceedings of Transactions of the Association for Computational Linguistics (TACL) | April 2024, pp. 247-263

Publication Project

Agent AI Towards a Holistic Intelligence

Qiuyuan Huang, Naoki Wake, Bidipta Sarkar, Zane Durante, Ran Gong, Rohan Taori, Yusuke Noda, Demetri Terzopoulos, Noboru Kuno, Ade Famoti, Ashley J. Llorens, John Langford, Hoi Vo, Fei-Fei Li, Katsushi Ikeuchi, Jianfeng Gao

arXiv: 2403.00833 | February 2024

Publication Publication

An Interactive Agent Foundation Model

Zane Durante, Bidipta Sarkar, Ran Gong, Rohan Taori, Yusuke Noda, Paul Tang, Ehsan Adeli, Shrinidhi Kowshika Lakshmikanth, Kevin Schulman, Arnold Milstein, Demetri Terzopoulos, Ade Famoti, Noboru Kuno, Ashley J. Llorens, Hoi Vo, Katsushi Ikeuchi, Fei-Fei Li, Jianfeng Gao, Naoki Wake, Qiuyuan Huang

February 2024

Publication Publication

Agent AI: Surveying the Horizons of Multimodal Interaction

Zane Durante, Qiuyuan Huang, Naoki Wake, Ran Gong , Jae Sung Park, Bidipta Sarkar, Rohan Taori, Yusuke Noda, Demetri Terzopoulos, Yejin Choi, Katsushi Ikeuchi, Hoi Vo, Fei-Fei Li, Jianfeng Gao

arXiv: 2401.03568 | January 2024

Publication

Logical Transformers: Infusing Logical Structures into Pre-Trained Language Models

Borui Wang, Qiuyuan Huang, Budhaditya Deb, Aaron L. Halfaker, Liqun Shao, Daniel McDuff, Ahmed Awadallah, Dragomir Radev, Jianfeng Gao

Proceedings of ACL 2023 | May 2023

Publication Publication Project

ArK: Augmented Reality with Knowledge Emergent Infrastructure

Qiuyuan Huang, James Park, Abhinav Gupta, Pan Lu, Paul Bennett, Ran Gong, Subhojit Som, Baolin Peng, Owais Khan Mohammed, Chris Pal, Yejin Choi, Jianfeng Gao

Selected as the project in HackBox 2023; Shipped to Office Teams | April 2023

Publication Project

NICE: Neural Image Commenting with Empathy

Kezhen Chen, Qiuyuan Huang, Daniel McDuff, Xiang Gao, Hamid Palangi, Jianfeng Wang, Kenneth Forbus, Jianfeng Gao

EMNLP 2021 | October 2021

Publication

Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation

Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Weng, William Yang Wang, Lei Zhang

CVPR 2019 | June 2019

Best Student Paper Award in CVPR 2019.

Publication Video Project Project

Turbo Learning for CaptionBot and DrawingBot

Qiuyuan Huang, Pengchuan Zhang, Oliver Wu, Lei Zhang

NeurIPS 2018 | December 2018

PDF Publication Project

LLM Agent – Retrieve What You Need: A Mutual Learning Framework for Open-domain Question Answering

Dingmin Wang, Qiuyuan Huang, Matthew Jackson, Jianfeng Gao

Proceedings of Transactions of the Association for Computational Linguistics (TACL) | April 2024, pp. 247-263

Publication Project

An Interactive Agent Foundation Model

Zane Durante, Bidipta Sarkar, Ran Gong, Rohan Taori, Yusuke Noda, Paul Tang, Ehsan Adeli, Shrinidhi Kowshika Lakshmikanth, Kevin Schulman, Arnold Milstein, Demetri Terzopoulos, Ade Famoti, Noboru Kuno, Ashley J. Llorens, Hoi Vo, Katsushi Ikeuchi, Fei-Fei Li, Jianfeng Gao, Naoki Wake, Qiuyuan Huang

February 2024

Publication Publication

MindAgent: Emergent Gaming Interaction

Steven Gong, Qiuyuan Huang, Xiaojian Ma, Hoi Vo, Zane Durante, Yusuke Noda, Zilong Zheng, Song-chun Zhu, Demetri Terzopoulos, Feifei Li, Jianfeng Gao

Processings of NAACL 2024 | January 2024

Publication Project

Agent AI: Surveying the Horizons of Multimodal Interaction

Zane Durante, Qiuyuan Huang, Naoki Wake, Ran Gong , Jae Sung Park, Bidipta Sarkar, Rohan Taori, Yusuke Noda, Demetri Terzopoulos, Yejin Choi, Katsushi Ikeuchi, Hoi Vo, Fei-Fei Li, Jianfeng Gao

arXiv: 2401.03568 | January 2024

Publication

Logical Transformers: Infusing Logical Structures into Pre-Trained Language Models

Borui Wang, Qiuyuan Huang, Budhaditya Deb, Aaron L. Halfaker, Liqun Shao, Daniel McDuff, Ahmed Awadallah, Dragomir Radev, Jianfeng Gao

Proceedings of ACL 2023 | May 2023

Publication Publication Project

ArK: Augmented Reality with Knowledge Emergent Infrastructure

Qiuyuan Huang, James Park, Abhinav Gupta, Pan Lu, Paul Bennett, Ran Gong, Subhojit Som, Baolin Peng, Owais Khan Mohammed, Chris Pal, Yejin Choi, Jianfeng Gao

Selected as the project in HackBox 2023; Shipped to Office Teams | April 2023

Publication Project

VLC: Training Vision-Language Transformers from Captions

Liangke Gui, Qiuyuan Huang, Alex Hauptmann, Yonatan Bisk, Jianfeng Gao

Proceedings of Transactions on Machine Learning Research | May 2022

Publication Project

KAT: A Knowledge Augmented Transformer for Vision-and-Language

Liangke Gui, Borui Wang, Qiuyuan Huang, Alex Hauptmann, Yonatan Bisk, Jianfeng Gao

NAACL 2022. Long paper, Oral. | January 2022

No.1 of OK-VQA Leaderboard (2021).

Publication

NICE: Neural Image Commenting with Empathy

Kezhen Chen, Qiuyuan Huang, Daniel McDuff, Xiang Gao, Hamid Palangi, Jianfeng Wang, Kenneth Forbus, Jianfeng Gao

EMNLP 2021 | October 2021

Publication

Mapping Natural-language Problems to Formal-language Solutions Using Structured Neural Representations

Kezhen Chen, Qiuyuan Huang, Hamid Palangi, Paul Smolensky, Kenneth D. Forbus, Jianfeng Gao

ICML 2020 | July 2020

Publication Project

Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation

Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Weng, William Yang Wang, Lei Zhang

CVPR 2019 | June 2019

Best Student Paper Award in CVPR 2019.

Publication Video Project Project

Hierarchically Structured Reinforcement Learning for Topically Coherent Visual Story Generation

Qiuyuan Huang, Zhe Gan, Asli Celikyilmaz, Oliver Wu, Jianfeng Wang, Xiaodong He

AAAI 2019 | January 2019

Publication Project Project

Turbo Learning for CaptionBot and DrawingBot

Qiuyuan Huang, Pengchuan Zhang, Oliver Wu, Lei Zhang

NeurIPS 2018 | December 2018

PDF Publication Project

An Interactive Agent Foundation Model

Zane Durante, Bidipta Sarkar, Ran Gong, Rohan Taori, Yusuke Noda, Paul Tang, Ehsan Adeli, Shrinidhi Kowshika Lakshmikanth, Kevin Schulman, Arnold Milstein, Demetri Terzopoulos, Ade Famoti, Noboru Kuno, Ashley J. Llorens, Hoi Vo, Katsushi Ikeuchi, Fei-Fei Li, Jianfeng Gao, Naoki Wake, Qiuyuan Huang

February 2024

Publication Publication

Agent AI: Surveying the Horizons of Multimodal Interaction

Zane Durante, Qiuyuan Huang, Naoki Wake, Ran Gong , Jae Sung Park, Bidipta Sarkar, Rohan Taori, Yusuke Noda, Demetri Terzopoulos, Yejin Choi, Katsushi Ikeuchi, Hoi Vo, Fei-Fei Li, Jianfeng Gao

arXiv: 2401.03568 | January 2024

Publication

Logical Transformers: Infusing Logical Structures into Pre-Trained Language Models

Borui Wang, Qiuyuan Huang, Budhaditya Deb, Aaron L. Halfaker, Liqun Shao, Daniel McDuff, Ahmed Awadallah, Dragomir Radev, Jianfeng Gao

Proceedings of ACL 2023 | May 2023

Publication Publication Project

ArK: Augmented Reality with Knowledge Emergent Infrastructure

Qiuyuan Huang, James Park, Abhinav Gupta, Pan Lu, Paul Bennett, Ran Gong, Subhojit Som, Baolin Peng, Owais Khan Mohammed, Chris Pal, Yejin Choi, Jianfeng Gao

Selected as the project in HackBox 2023; Shipped to Office Teams | April 2023

Publication Project

Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation

Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Weng, William Yang Wang, Lei Zhang

CVPR 2019 | June 2019

Best Student Paper Award in CVPR 2019.

Publication Video Project Project

Agent AI Towards a Holistic Intelligence

Qiuyuan Huang, Naoki Wake, Bidipta Sarkar, Zane Durante, Ran Gong, Rohan Taori, Yusuke Noda, Demetri Terzopoulos, Noboru Kuno, Ade Famoti, Ashley J. Llorens, John Langford, Hoi Vo, Fei-Fei Li, Katsushi Ikeuchi, Jianfeng Gao

arXiv: 2403.00833 | February 2024

Publication Publication

An Interactive Agent Foundation Model

Zane Durante, Bidipta Sarkar, Ran Gong, Rohan Taori, Yusuke Noda, Paul Tang, Ehsan Adeli, Shrinidhi Kowshika Lakshmikanth, Kevin Schulman, Arnold Milstein, Demetri Terzopoulos, Ade Famoti, Noboru Kuno, Ashley J. Llorens, Hoi Vo, Katsushi Ikeuchi, Fei-Fei Li, Jianfeng Gao, Naoki Wake, Qiuyuan Huang

February 2024

Publication Publication

MindAgent: Emergent Gaming Interaction

Steven Gong, Qiuyuan Huang, Xiaojian Ma, Hoi Vo, Zane Durante, Yusuke Noda, Zilong Zheng, Song-chun Zhu, Demetri Terzopoulos, Feifei Li, Jianfeng Gao

Processings of NAACL 2024 | January 2024

Publication Project

Agent AI: Surveying the Horizons of Multimodal Interaction

Zane Durante, Qiuyuan Huang, Naoki Wake, Ran Gong , Jae Sung Park, Bidipta Sarkar, Rohan Taori, Yusuke Noda, Demetri Terzopoulos, Yejin Choi, Katsushi Ikeuchi, Hoi Vo, Fei-Fei Li, Jianfeng Gao

arXiv: 2401.03568 | January 2024

Publication

Multimodal Agent — Localized Symbolic Knowledge Distillation for Visual Commonsense Models

Jae Sung Park, Jack Hessel, Khyathi Chandu, Paul Pu Liang, Ximing Lu, Peter West, Qiuyuan Huang, Jianfeng Gao, Ali Farhadi, Yejin Choi

Proceedings of NeurIPS 2023 | October 2023

Publication

Logical Transformers: Infusing Logical Structures into Pre-Trained Language Models

Borui Wang, Qiuyuan Huang, Budhaditya Deb, Aaron L. Halfaker, Liqun Shao, Daniel McDuff, Ahmed Awadallah, Dragomir Radev, Jianfeng Gao

Proceedings of ACL 2023 | May 2023

Publication Publication Project

ArK: Augmented Reality with Knowledge Emergent Infrastructure

Qiuyuan Huang, James Park, Abhinav Gupta, Pan Lu, Paul Bennett, Ran Gong, Subhojit Som, Baolin Peng, Owais Khan Mohammed, Chris Pal, Yejin Choi, Jianfeng Gao

Selected as the project in HackBox 2023; Shipped to Office Teams | April 2023

Publication Project

KAT: A Knowledge Augmented Transformer for Vision-and-Language

Liangke Gui, Borui Wang, Qiuyuan Huang, Alex Hauptmann, Yonatan Bisk, Jianfeng Gao

NAACL 2022. Long paper, Oral. | January 2022

No.1 of OK-VQA Leaderboard (2021).

Publication

NICE: Neural Image Commenting with Empathy

Kezhen Chen, Qiuyuan Huang, Daniel McDuff, Xiang Gao, Hamid Palangi, Jianfeng Wang, Kenneth Forbus, Jianfeng Gao

EMNLP 2021 | October 2021

Publication

Mapping Natural-language Problems to Formal-language Solutions Using Structured Neural Representations

Kezhen Chen, Qiuyuan Huang, Hamid Palangi, Paul Smolensky, Kenneth D. Forbus, Jianfeng Gao

ICML 2020 | July 2020

Publication Project

Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation

Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Weng, William Yang Wang, Lei Zhang

CVPR 2019 | June 2019

Best Student Paper Award in CVPR 2019.

Publication Video Project Project

Hierarchically Structured Reinforcement Learning for Topically Coherent Visual Story Generation

Qiuyuan Huang, Zhe Gan, Asli Celikyilmaz, Oliver Wu, Jianfeng Wang, Xiaodong He

AAAI 2019 | January 2019

Publication Project Project

Turbo Learning for CaptionBot and DrawingBot

Qiuyuan Huang, Pengchuan Zhang, Oliver Wu, Lei Zhang

NeurIPS 2018 | December 2018

PDF Publication Project

LLM Agent – Retrieve What You Need: A Mutual Learning Framework for Open-domain Question Answering

Dingmin Wang, Qiuyuan Huang, Matthew Jackson, Jianfeng Gao

Proceedings of Transactions of the Association for Computational Linguistics (TACL) | April 2024, pp. 247-263

Publication Project

VLC: Training Vision-Language Transformers from Captions

Liangke Gui, Qiuyuan Huang, Alex Hauptmann, Yonatan Bisk, Jianfeng Gao

Proceedings of Transactions on Machine Learning Research | May 2022

Publication Project

Vision-Language Navigation Policy Learning and Adaptation

Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-fang Wang, William Wang, Lei Zhang

IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2020 | February 2020

Publication Publication Project