Hello FIFA fans. I’m Joel Doonan, a Live Producer on FIFA. In this Pitch Notes article I am going to talk a little bit about FIFA Game Data Centers (FGDC) in FIFA 20. Specifically around the process of how we actually evaluate and add a new FGDC location to our existing infrastructure.

In our recent FIFA Game Data Center FAQ Pitch Notes article, we covered a number of topics, including a summary explanation around how we decide where to place our FGDCs. Everything in that article remains accurate today, however, we saw some of the questions that arose from the community after we published that article, and wanted to go into more detail around the process to eliminate any confusion you may have.

I am also going to dig into the recent addition of a FGDC in Dallas to help illustrate some of the improvements that we have seen with that location being added, along with how it serves as a great example of why adding FGDC locations isn’t always a seemingly logical science.

So let’s get at it!

Adding FGDC Locations For FIFA

In the first part of this article, I am going to detail the four phases that we run through when looking to add an additional FGDC.

Those Phases are:

• Identifying potential new FGDC locations.
• Measuring the potential impact of a new FGDC location.
• Proof of concept for a new FGDC location.
• Official launch of a new FGDC location.

Identifying Potential FGDC Locations

Phase 1 TL/DR: 

• We use data to identify where adding a FGDC could have a positive impact to a player community.
• We look to identify a vendor or partner in that area that we can work with.
  • If none are available we probably need to wait until there are.
• If we find a vendor/partner and the business aspects align, we proceed to the measuring the potential impact of a new FGDC phase.

The first step in identifying a potential new location for a FGDC starts with looking at data. We look at a bunch of super interesting (to us at least) data, graphs and heat maps that help us identify when we have locations around the world where a player community is currently not hitting the internal targets that we have for our networking metrics, such as their average ping.

After identifying a region or location that we want to try to improve, we need to confirm whether or not there is an experienced and reliable vendor in that area that we can partner with. As we covered in the FIFA Game Data Center FAQ Pitch Notes, if we set up a FGDC with a vendor that doesn’t meet that criteria, the potential for reliability and stability issues that would impact the player experience would be much higher. That’s not something we are ok with.

However, this can be a challenge as there are still fairly large areas of the world where the quality of vendors that we require don’t yet have a presence. This means that sometimes this is the step where we are forced to press pause, waiting until one of these vendors establishes a presence in or around that location.

If, however, we identify that we do have a vendor/partner that could service our needs in that area, and all the appropriate business hurdles are cleared, we would move onto the next stage of this process, measuring the potential impact of a new FGDC.

Measuring The Potential Impact Of A New FGDC Location

Phase 2 TL/DR:

• Before we add a FGDC location we need to confirm, with data, that it is actually going to meaningfully improve the online gameplay experience for enough players.
• Physical proximity to a FGDC location doesn’t always equal a good network connection as the internet often doesn’t work that way.
• During this phase, regardless of what you may see on social media, the FGDC is not yet officially live.
  • Official announcements come from @EAFIFADirect on Twitter.
• If the data gathered indicates a meaningful improvement, and there are no other blockers, we move onto the next phase, proof of concept.

Once we have decided that we want to potentially add a new FGDC to a location, and we have identified a vendor/partner that we can work with, the next step is to gather data on how that FGDC could improve the online gameplay experience for our players. 

Now, this is often the point where a more detailed understanding of how the internet works is key, as without that understanding, it’s reasonable to assume that it would be obvious that adding a FGDC to a location would improve the online gameplay experience for players in that location. Unfortunately, that isn’t always the case.

To help explain, let’s go on a brief educational detour for a second, as it’s important to understand a little about how the internet works to fully understand why.

When playing a FIFA Online Match, what is happening on the virtual pitch relies on game data being sent back and forth over the internet between you and whatever FGDC you are playing on. When that data is travelling from one place to another, it will travel along an internet route until it reaches its final destination. The route that it takes is determined by a number of factors, such as your Internet Service Provider (ISP), the relationships that your ISP has made with other ISPs, what internet backbone it is using, etc. Ultimately, you have little direct control over that route, nor would you likely be aware of the route that it’s taking, unless you were actively trying to monitor for this, and probably wouldn’t think to care much about it either. However, the routing taking place here can be critical to a positive online gameplay experience in FIFA (or any other game for that matter), and here is why.

Let’s say that you, totally hypothetically, live in Vancouver, Canada. You are playing a FIFA Online Match at home. The physically closest FGDC to where you live would be the location in Portland, USA, and it’s reasonably close, so you would expect that you would normally be playing your matches in that location. 

However, let’s say that your ISP has partnered with another ISP in the US to use their backbone networking infrastructure, and that their main networking hub is in Chicago, which means that the first place your data goes after it leaves your house is Chicago, no matter where the data needs to eventually end up.

Now if you look at a map of the US, Chicago isn’t anywhere near Portland, in fact it’s much closer physically to the FGDC in Columbus, USA. This means that it’s quite likely that the ping to get to the FGDC in Columbus is now lower than the ping it would take to get to the one in Portland. This means, that in all likelihood, you are now going to be playing most of my matches in Columbus instead, because it would actually provide you with the lowest ping and fastest connection in these circumstances. It’s likely that a direct route from Vancouver to Portland would have been a lower ping, but in this example, that route isn’t being used by your ISP, so the Vancouver to Columbus route actually ends up being the best option available for you.

Now situations like this hypothetical example are happening all over the world, all of the time. Factors such as ISPs and internet backbone business decisions, political situations, local infrastructure, network outages or maintenance and many others contribute to situations where the route that data takes over the internet is often not reflective of what someone would think is logical. To make things more complicated, sometimes the impact is very significant, like data from one continent being sent to another continent, across an entire ocean, before it could route back to the continent of origin. This isn’t a hypothetical example, we have seen it first hand on many occasions.

So, with the above in mind, you can likely see where this is going. Before we officially add and launch a FGDC, we need to ensure that if we are setting something up in a new location that the players in and around that location are actually going to be able to connect to it efficiently. If they don’t, they may end up still playing on a FGDC in another location, essentially defeating the purpose of adding that new location entirely.

Given the very unique networking situations that each of our players may have, we are pretty limited in how much we can guess how it is going to work for players. To really understand what is going to happen we need to set up something in that potential new FGDC location in order to get data on what players' connection quality to that location actually is. 

Now, when this happens, those players mentioned earlier, who may be using tools to monitor their network/internet, are going to be able to see data going to this new potential FGDC location. We have actually seen some instances of players sharing this information on Twitter or other sites, assuming that it meant that a new FGDC had been added. In actuality, we were still in the potential impact measurement stage of a potential new rollout and no actual games are being played at this potential FGDC location. It’s key to understand that until we actually message about a new FGDC being added to the game, primarily through @EAFIFADirect on Twitter, that it’s not officially live. More on that later in the article.

The process of gathering this information may be pretty short, potentially a few weeks, or it could take a longer period of time, upwards of many months, depending on how many issues we may encounter during this process or how clear cut the data that we are getting back is.

If the investigation doesn’t show us that adding a FGDC to that location would meaningfully improve the experience for our players, then we likely won't move onto the next phase. Instead we may go back to the previous phase, investigating if a different vendor or partner may be able to give us a better solution for that area of the world.

If the investigation does indicate that this new location would meaningfully improve the online gameplay experience for our players, and we have satisfied any other requirements that would prevent us from moving forward, then it’s time to start setting up and rolling out the FGDC location proof of concept phase.

Proof Of Concept Phase

Phase 3 TL/DR:

• Time is required to prepare a FGDC for player use.
• A FGDC is soft launched first, to ensure we see stability, high performance and reliability before we officially launch it.
• A FGDC may not be at full capacity at the start of this phase, but would be at full capacity before this phase ended.
• During this phase, regardless of what you may see on social media, the FGDC is not yet officially live.
  • Official announcements come from @EAFIFADirect on Twitter.
• If everything checks out, we move onto the next phase, the official launch.

The previous phase gave us theoretical data about the potential new FGDC location, including an approximation of how many players would likely play their matches at this location based on the quality of their network connection to it. This information is what allows us to make the decision to move forward with this new location based on the improvements we think it can provide to our players.

What it can’t tell us is how well this FGDC will actually perform in terms of stability, performance or reliability. In order to understand that, we actually need to start setting up the new FGDC to start having FIFA Online Matches played on it, so we can monitor how it performs and ultimately make a final decision on whether we want to officially add this location. 

This is our proof of concept phase, which involves working with the vendor/partner to get hardware in place, and to do all the technical work that is required for the game to be able to connect to, and use, the new FGDC location.

To note, this is another spot where we will sometimes experience a delay, as, depending on the vendor, physical location and type of hardware that is required, actually getting it ready for use can take anywhere from a few weeks, to many months.

Once everything is ready, we will go into what is essentially a soft launch of the location.

During this soft launch, some players will be playing matches using the new FGDC. As that happens, we are monitoring the location very closely to ensure that it is performing as expected but it is not yet ready to be considered fully live. 

During the start of this phase it’s also possible that we haven’t scaled up the FGDC to full capacity yet, as the goal here is to prove out the new FGDC location before we fully commit to making it officially live. Part of the exit criteria for this phase would include completing that full build out in order to support the expected capacity of players.

If we are seeing any stability, performance or reliability issues, we may disable the location while we work to either resolve those issues, or reevaluate the plan completely, including possibly not moving forward with the location at all.

As mentioned in the previous phase, while this is happening, some players will be able to see some of these changes impacting where their network data is going, and they may start to talk about how there is a new FGDC location available. Again, until something has been announced through @EAFIFADirect on Twitter, it’s not official and it’s possible that the site may not officially go live in the short term, or potentially ever. Also, the length of this soft launch period could vary significantly, again ranging from weeks to months depending on what we are seeing during this initial monitoring period. 

Assuming that everything during the soft launch goes as hoped, we are ready to move into the final phase, which is officially announcing the location and moving it into our full operational processes.

FGDC Location Launch

Phase 4 TL/DR:

• We officially announce the FGDC via@EAFIFADirect on Twitter

• It’s now fully live for players and treated in the same manner as all previously existing FGDC locations.

We have arrived at the final step of adding a new FGDC location, which is to officially announce to our players that it is live. As mentioned previously, this announcement would happen via @EAFIFADirect on Twitter, though it may also be posted in other locations as well.

Once this happens, this means that this FGDC location will now be handled in the same manner as the other existing locations, where we are monitoring and evaluating them for stability, performance, reliability and capacity on a continuous basis. That evaluation may result in us making changes to our existing suite of FGDCs in the future.

And that wraps up the deep dive into the high level process we follow when looking to potentially add a new FGDC. Obviously every situation is unique, with its own timeline, challenges and requirements, but this should give you good insight into what goes on.

Of course, anytime we do this, we are ultimately doing it with the goal of improving the experience for our players, so let’s now shift gears and take a look at the recent addition of our FGDC in Dallas, Texas and what impact it has had.

Impact Of New Dallas FGDC

We announced that our Dallas FGDC location was officially live back in June 11th, 2020, but the process for adding that location began well before that, as we worked through the steps detailed in this article to identify whether or not this was the right place to set up a new location. The soft launch phase, which started in late May, was when we first started to get concrete data from our players about the impact that the Dallas FGDC was going to have.

First off, I want to preface the data I am about to go through here with some details around one of our goals. With Online Gameplay Responsiveness being a key focus for our team, we have a number of internal targets that we are striving to hit related to network metrics. One key network metric is the concept of a player’s ping, which is the measurement of how long it takes for a packet of game data to be sent, from your console or PC to the Game Data Centers and back. 

While many factors contribute to an individual player’s ping, including, but not limited to, their personal network setup, their ISP and their hardware, one of the factors is the distance between you and the FGDC that you are playing your match on, though it’s not just the physical distance, as we covered earlier in the article, but also the route that your data takes when travelling along the internet.

Currently, we are working towards an internal target of having as many FIFA Online Matches as possible being played with an average of 50ms of ping or less. While this isn’t a line in the sand for what constitutes an ok ping value versus a non-ok ping value, as we have feedback from players that games at a higher ping value can be responsive, we want to push an aggressive target, given our worldwide player population, as part of our efforts around Online Gameplay Responsiveness. So when we are evaluating various player communities around the world, this is one of the razors we are using to measure success.

So given that goal, let’s take a look at what adding a FGDC in Dallas did to move the needle against that target.

Given the physical location of Dallas, there are two primary countries that we would expect to see impacted in a positive manner by this addition, that being the United States and Mexico. 

Let’s start by looking at the United States. 

Starting at the full country level, when using the key measurement of the percentage of FIFA Online Matches that are being played under 50ms, you can see an approximately 5-7% improvement after the Dallas FGDC was launched.

When looking at the state of Texas on it’s own, you see a much more pronounced improvement. The percentage of FIFA Online Matches being played below 50ms went from between ~50-60% before the addition of the Dallas FGDC to between ~85-90% after.

So in summary, for the United States, this addition played out in a pretty logical manner. The players physically closest to the FGDC saw an improvement to their ping when playing matches. Overall, we are happy with how this FGDC location improved the experience for our players in the United States.

Now let’s talk about Mexico, because the story here is a little more unique and interesting and really helps to showcase how the internet often doesn’t align with expectations based off physical locations.

To start, let’s look at Mexico at the country level. Prior to the addition of the Dallas FGDC, we were sitting at just under the ~20% mark for FIFA Online Matches being played below 50ms. With Dallas live, that value increased significantly, sitting for the most part at around ~45%.

Now initially, that may not seem too odd. Dallas, Texas is in fairly close proximity to the United States and Mexico border, and it’s reasonable to assume that, despite us having a FGDC in Mexico City, that the northern parts of the country may have a better connection to Dallas than to Mexico City.

However, when looking at this heat map, which shows a breakdown of what percentage of FIFA Online Matches were being played on the new Dallas FGDC, we see something pretty interesting.

While we do see a fairly high percentage of matches in the northern parts of Mexico being played on the Dallas FGDC, we also see the majority of matches from Mexico City, and some surrounding areas, being played on Dallas as well, despite there being a FGDC directly in Mexico City itself.

If you focus in on Mexico City, you can see the impact that the Dallas FGDC location had on the players in that location, with the percentage of matches being played under 50ms increasing significantly.

It’s key to note however, when looking at the heat map again, that this isn’t a case of the Mexico City FGDC underperforming for all players. There are still significant areas in Mexico whereby their best connection was not to the Dallas FGDC, but to the one in Mexico City.

So we have a situation where the people living in a city, potentially due to the internet routing challenges that were outlined earlier in the article, are able to maintain a faster connection to a FGDC location in another country than they can to the one located within the city that they live in.

And that really is the challenge of trying to use FGDCs to improve the Online Gameplay Responsiveness for FIFA players in a nutshell. Without a detailed understanding of the internet, it may seem simple, and sometimes it can be, but often you are faced with realities such as these. 

Getting it right, and building out FGDCs that properly support players is much more complicated than sticking a bunch of pins on a map. Making meaningful improvements relies on the availability of appropriate infrastructure, solid data analysis and often, trial and error.

We are very happy with the impact that the Dallas FGDC has had on the players that are now playing their FIFA Online Matches on it, the data shows that it’s made a positive difference for many players, which is always our goal.

Hopefully this deep dive has been informative, and helps to showcase both the efforts that we are putting into this space, as well as why it’s not always as simple as it seems like it could be, or ultimately we wish it was. This continues to be a major focus for us, and we are committed to keeping at it. As mentioned earlier, make sure you are following @EAFIFADirect on Twitter for any updates about FGDC locations going live, new Pitch Notes articles and for any other information from the FIFA Development Team.

Thanks again for reading, as always, we appreciate your support and passion for the game.

Joel Doonan and the FIFA Team

--

For more deep dives on FIFA from members of the game team, check out the Pitch Notes Page.

Please Note: This article is describing in general terms what updates the Dev Teams are working for FIFA 20 on PlayStation 4, Xbox One and PC. We are constantly looking to improve the FUT experience for everyone, so this article may become outdated as we make adjustments to keep our game fun for everyone.

FIFA 20 is available now for PlayStation 4, Xbox One, and PC.

Stay in the conversation on all things FIFA by liking us on Facebook , following us on Twitter and Instagram , and participating in the official FIFA Forums . Sign-up to receive emails about EA SPORTS FIFA and EA products, news, events, and promotions.