Google console fetch & render does display AJAX fetched content - ajax

Question

I have published a reactjs website that relies on AJAX requests (POST requests on a graphQL API if that's relevant) to display data.
Using google console fetch & render service, I can see that only my components that do not have to call any API are rendered. Any AJAX based component is not rendered at all.
Google fetch & render does show me 2 rendering pictures of my website (google vs visitor), but both are missing AJAX content.
Is server rendering mandatory in this case ?
I do not have a robots.txt file.
I'm doing something like:
import React, { Component } from 'react';
import { observer } from 'mobx-react';
import { observable, runInAction } from 'mobx';
import axios from 'axios';
import ContributorList from '~/components/global/ContributorList';
import type { User } from '~/types';
import CSSModules from 'react-css-modules';
import styles from './style.less';
#observer
#CSSModules(styles)
export default class Footer extends Component {
#observable contributors: Array<User> = [];
async loadContributors () {
const { data: {
data: data
} } = await axios.post('/', {
query: `
{
users {
id,
name,
slug,
avatar {
secure_url
}
}
}
`
});
runInAction( () => {
this.contributors = data.users;
});
}
componentDidMount () {
this.loadContributors();
}
render () {
return (
<div styleName='contributors'>
{ 'Static content' }
<ContributorList
contributors={ this.contributors }
/>
</div>
);
}
};
In my browser, I can see everything fine ('Static content' + contributors that rare fetched asynchronously). But with google fetch and render, I see 2 screenshots with only 'Static content' displayed.
As a result, my dynamic content does not appear in google search results.

Answer 1

Now all the people are talking about progressive web apps (PWA) which rely on Ajax and rendering the content of the website using client side techniques.
But these techniques are not SEO friendly at all, since Google can't index the classic Ajax request and most of the modern requests.
Since you are using ReactJs, and you want Google to index your website you have to use server side rendering.
This will slow down your website and at the same time Google can crawl and index all your pages, also the user can view it on any device, even the old devices.
When you want to build a PWA or modern web app in general, you have to go back to the basics, for an old technique called graceful degration, which mean if you disabled the JS from your browser still you can see the content and all the functions on your website are work.
This is the same way Google crawl and index your website.
There are many recommendations from Google regarding crawling Ajax content:
Use clean URL structure and avoid using JS links(javascript:void(0);)
Load all the using server side rendering
Make sure that your website is working on all devices(responsive)
Submit XML sitemap
Use canonical links in if you have more than one URL structure (which is not recommended)
We are working on a new project similar to yours, built using reactjs and it will be one of the first PWAs in the world that Google can crawl it like any other website by doing the above points.
Also Google have published an article on their webmaster blog talking about the PWA and any modern web app like yours.
For more details check this link
https://webmasters.googleblog.com/2016/11/building-indexable-progressive-web-apps.html
Regarding the robots.txt file, its recommended to add robots for any website even if you don't have pages to block, but you may want to block the ajax requests and other annoying crawlers.