Who’s Ahead in AI Research? Insights from NIPS, Most Prestigious AI Conference

Please subscribe to our Twitter if you’d like to follow our Deep Tech research. 🤖

Despite what the media might make you think, the West is far, far ahead in Artificial Intelligence research, and there is no imminent threat of China taking over. We know this from our deep dive into the most prestigious international AI research venue, the Conference on Neural Information Processing Systems (NIPS). The latest one, NIPS 2017, was held in Long Beach, California, attracting over 8,000 registered attendees, up 2,000 from the prior year. 679 papers out of 3,240 submitted were accepted for a 21% acceptance rate. Using these conference proceedings, we went into each of the 679 accepted papers and compiled a complete list of 2,497 authors and their affiliated organizations (many repeating, of course), and then calculated something we called the Publication Index.

What you can see below is the resulting chart roughly following the Power Law distribution with a factor of 3: the United States is in clear lead with a Publication Index of 414, followed by Western Europe with an index of 136, and China being the distant third, with an index of 39. (As an aside, we define Western Europe as the EEA (European Economic Area) + Switzerland, where the EEA includes the European Union, Norway, and Luxembourg; we feel that it’s fair to bundle these European countries together due to healthy coordination of research funding and multinational cooperation between them.)

Top 10 regions that published AI research at NIPS 2017

Here’s how the Publication Index we created works: each publication is assigned one point, which is then divided evenly across its N authors, 1/N each (we assumed equal contribution as the first-order approximation). We then assign these points to each author’s primary affiliated organization (sometimes there’s a secondary or even tertiary affiliation that we ignore for this study). For example, if a paper has five authors — three from MIT, one from the University of Oxford, and one from Google — each author will get 1/5th of one point, or 0.2. As a result, from this paper alone, MIT will increase its Publication Index by 3*0.2=0.6 points, the University of Oxford will increase its index by 0.2, and Google will add 0.2. Since MIT is based in the United States, MIT affiliation will increase the Publication Index of the United States by 0.6. Similarly, since the University of Oxford is based in the UK, the EEA + Switzerland category will increase by 0.2. Finally, Google is a multinational corporation headquartered in the United States, therefore the United States will increase its Publication Index by an additional 0.2, for the total increase of 0.8. The idea here is to establish a consistent methodology that allocates credit in inverse proportion to the number of authors in a publication, which by construction should produce a set of fair aggregate statistics.

At this point, you might be interested in the rankings by specific country, rather than in aggregate:

Top 10 countries leading in AI research at NIPS 2017

The results here are even more dramatic, with the United States dominating the AI research with the Publication Index of 414 (just as before), but the next three players are an order of magnitude smaller, with China at 39, France at 37 and the UK at 34. In other words, the United States has a 10x lead on China when it comes to publishing advanced AI research.

What else did we discover when we looked at the data? If you were to guess the top five global leaders in AI research, across both academia and industry, who would they be?

Top 25 global organizations (industry and academia) leading in AI research, based on publications at NIPS 2017

The US also holds top places in this category. No surprise that Google, with its DeepMind, Google Brain, and Google Research divisions and access to petabytes of consumer and business data, is the powerhouse here, at first place. Then come the four dream graduate schools for any aspiring AI and robotics Ph.D. student — Carnegie Mellon University (CMU), Massachusetts Institute of Technology (MIT), Stanford University and UC Berkeley, with the second, third, fourth, and fifth places.

Next, let’s take a look at how academia is fairing versus the industry in its fight for brilliant AI researchers:

Academia vs. industry in AI research at NIPS 2017

It is interesting to see that almost a fifth of AI research is now coming from the industry. Gone are the days when graduating Ph.D. students, postdocs, and principal investigators had to forego all hopes for publishing their work when going to work for the “dark side”. This is a big deal, and it’s encouraging that AI research community is insisting on keeping research results open. The battle is not won yet, as we didn’t see any research from Apple published at NIPS 2017 while Apple is one of the key players in the space, with its Siri app and HomePod products.

Since we are on the subject of corporations, let’s see how they stack up against each other:

Top 20 global companies leading AI research at NIPS 2017

We had already expected Google here as the number one, of course. Microsoft, with its elite Microsoft Research unit, is in second place, and Facebook, with FAIR, is in third. IBM, with Watson, holds the fourth place. Toyota, thanks to Toyota Research Institute, is at number five.

The last question we’ll address here is where you should go to grad school if you want to surround yourself with top AI researchers, at least based on what we know from NIPS. Starting with schools in the United States only:

Top 20 American universities leading in AI research at NIPS 2017

Now, if you are open to going to grad school anywhere in the world, here is the top 25 chart:

Top 25 global universities leading in AI research at NIPS 2017

With that, we conclude our analysis of NIPS 2017. There are still many open questions, of course. For instance, you might ask if China is just being secretive about their latest AI research, and that’s the reason there aren’t more papers published by Chinese universities and companies. While this is definitely a possibility, we tend to think it’s an unlikely one. After all, getting published at NIPS is a ticket to global employment opportunities for any Chinese AI researcher (assuming that’s of value). The more likely explanation is that China is behind, and their AI strategy amounts to copying research results from elsewhere, and then applying them to domestic datasets. To quote Peter Thiel’s Zero to One, “The Chinese have been straightforwardly copying everything that has worked in the developed world: 19th-century railroads, 20th-century air conditioning, and even entire cities. They might skip a few steps along the way — going straight to wireless without installing landlines, for instance — but they’re copying all the same.” The Center for a New American Security in their Strategic Competition in an Era of Artificial Intelligence report tends to agree as well:

“During the last year of the Obama administration, the White House released several papers designed to move the United States toward a more coherent approach to artificial intelligence. Covering issues ranging from regulation to innovation to bias, these reports drove a series of conversations between scientists and government officials. Some of the authors of this report have argued that China’s AI strategy reflects the key principles from the Obama administration report — now it is China adopting them, instead of the United States.”

NIPS 2018 is coming to Montréal in December, so we should be able to update this analysis soon. If you are interested in AI research, the other conference that you should probably keep an eye on is the International Conference on Machine Learning (ICML). It is the second most prestigious international AI research conference, with a 25.1% acceptance rate in 2018. We’ll run a similar analysis on the proceedings from ICML 2018 in one of the next posts. If you can’t wait, Robbie Allen did a nice job analyzing ICML 2018 here (you’ll notice that his methodology is different, but high-level conclusions should still hold).

Hope you had as much fun reading this as we had researching NIPS for this study. More research is coming soon, so please check back often. Thank you for your time, and please leave any feedback/suggestions in the comments section. Cheers!

About me: My name is Gleb Chuvpilo, and I’m the Managing Partner at Thundermark Capital, a Venture Capital firm that invests in AI and robotics startups. I have a Master’s degree from the MIT Computer Science and Artificial Intelligence Lab and an MBA in Finance and Strategic Management from The Wharton School at the University of Pennsylvania. You can read more about me here. Please drop me a line at gleb@thundermark.com if you’d like to talk about AI, robotics, innovation in general, or your startup idea in particular. 🤖

PS: Here are all the rankings we discussed, this time in plain text:

Top 10 regions that published AI research at NIPS 2017:

1. USA
2. EEA + Switzerland
3. China
4. Japan
5. Canada
6. Israel
7. Korea
8. Australia
9. Singapore
10. India

Top 10 countries leading in AI research at NIPS 2017:

1. USA
2. China
3. France
4. UK
5. Japan
6. Canada
7. Israel
8. Switzerland
9. Germany
10. Finland

Top 25 global organizations (industry and academia) leading in AI research, based on publications at NIPS 2017:

1. Google (USA)
2. Carnegie Mellon University (USA)
3. Massachusetts Institute of Technology (USA)
4. Stanford University (USA)
5. UC Berkeley (USA)
6. Microsoft (USA)
7. University of Illinois at Urbana-Champaign (USA)
8. Inria (France)
9. ETH Zurich (Switzerland)
10. Duke University (USA)
11. University of Toronto (Canada)
12. Princeton University (USA)
13. University of Cambridge (UK)
14. Georgia Institute of Technology (USA)
15. University of Oxford (UK)
16. EPFL (Switzerland)
17. University of Michigan (USA)
18. New York University (USA)
19. Harvard University (USA)
20. Columbia University (USA)
21. Tsinghua University (China)
22. Cornell University (USA)
23. Technion (Israel)
24. University of Southern California (USA)
25. Facebook (USA)

Top 20 global companies leading AI research at NIPS 2017:

1. Google (USA)
2. Microsoft (USA)
3. Facebook (USA)
4. IBM (USA)
5. Toyota (Japan)
6. Adobe (USA)
7. Amazon (USA)
8. NTT (Japan)
9. OpenAI (USA)
10. NEC (Japan)
11. Disney (USA)
12. Tencent (China)
13. Mitsubishi (Japan)
14. Curious AI Company (Finland)
15. prowler.io (UK)
16. Nokia (Finland)
17. NVIDIA (USA)
18. Baidu (China)
19. Intel (USA)
20. Salesforce (USA)

Top 20 American universities leading in AI research at NIPS 2017:

1. Carnegie Mellon University (CMU)
2. Massachusetts Institute of Technology (MIT)
3. Stanford University
4. UC Berkeley
5. University of Illinois at Urbana-Champaign
6. Duke University
7. Princeton University
8. Georgia Institute of Technology
9. University of Michigan
10. New York University
11. Harvard University
12. Columbia University
13. Cornell University
14. University of Southern California
15. UT Austin
16. University of California, Los Angeles (UCLA)
17. UC San Diego
18. University of Wisconsin-Madison
19. University of Massachusetts Amherst
20. University of Washington

Top 25 global universities leading in AI research at NIPS 2017:

1. Carnegie Mellon University (USA)
2. Massachusetts Institute of Technology (USA)
3. Stanford University (USA)
4. UC Berkeley (USA)
5. University of Illinois at Urbana-Champaign (USA)
6. Inria (France)
7. ETH Zurich (Switzerland)
8. Duke University (USA)
9. University of Toronto (Canada)
10. Princeton University (USA)
11. University of Cambridge (UK)
12. Georgia Institute of Technology (USA)
13. University of Oxford (UK)
14. EPFL (Switzerland)
15. University of Michigan (USA)
16. New York University (USA)
17. Harvard University (USA)
18. Columbia University (USA)
19. Tsinghua University (China)
20. Cornell University (USA)
21. Technion (Israel)
22. University of Southern California (USA)
23. UT Austin (USA)
24. University of California, Los Angeles (USA)
25. UC San Diego (USA)

About the author: My name is Gleb Chuvpilo, and I’m the Managing Partner at Thundermark Capital, a Venture Capital firm that invests in Deep Tech startups. I have a Master’s degree from the MIT Computer Science and Artificial Intelligence Lab and an MBA in Finance and Strategic Management from The Wharton School at the University of Pennsylvania.