“It doesn’t exist, so let’s make it!” Why Mercari developed its Go Bold English & Japanese speaking tests

Hi, this is ＠JohnV from the Language Education Team (LET).

As the name suggests, LET focuses on designing and managing Mercari’s English and Japanese training programs. In addition to that, we are leading the Yasashii Communication initiative among other language and communication projects.

One of those projects is the development of our in-house communication tests – the Mercari English Communication Test (MECT) and the Mercari Japanese Communication Test (MJCT).

These tests serve as the foundation of all of our work, and were actually one of the first things we created when the team was established in 2018.

We need the tests to understand people’s initial level, as well as to track their progress through the training programs. The scores are also a key data point for the Yasashii Communication training, since a certain level of ability is required to make Yasashii Nihongo/Eigo (Easy & Kind Japanese/English) effective.

Why did LET make its own tests?

A question people sometimes ask is why did LET make its own tests when there are already-made tests available. And yes, a large number of companies in Japan use the traditional options for evaluating English and Japanese. However, after looking at these and trying out a lot of the tests, none of them matched Mercari’s needs.

First of all, many of the tests struggle to measure what learners can actually do using the language they are learning. More often the score represents the person’s passive knowledge of grammar or vocabulary rather than the ability to communicate. Of course passive knowledge is important, but we wanted to place a higher priority on measuring active speaking skills.

In addition, for the majority of the tests available at that time there was no clear way to correlate Person A’s English proficiency score and Person B’s Japanese proficiency. In fact, looking at people’s scores did not allow anyone to judge if they would be able to communicate with each other, which is important when selecting interviewers for hiring, for example, or making other personnel decisions.

Taking these and other factors into consideration, LET decided it was necessary to make our own tests. From here, I’d like to explain the benefits of the MECT and MJCT in a little more detail based on Mercari’s values (Go Bold: Let’s act boldly, All for One: All for success, and Be a Pro: Be professional).

Go Bold: Using the CEFR as the internal standard for both Japanese & English, which is nearly unprecedented in Japan

Rather than using the somewhat “local” solutions here in Japan, LET wanted to use the global standard as the scoring framework for our tests. Specifically, we decided to use the Common European Framework of Reference for Languages (CEFR), which breaks down language ability on a six-point scale, from A1 for beginners, up to C2 for those who have mastered a language. One interesting point is that it uses the same scale and level descriptions for every language.

With its widespread adoption in Europe, the CEFR is now being used by schools and companies across the world to evaluate language abilities because of its system of “can do” descriptors at each level. These statements clearly describe what a person at each level can do in any language.

This was particularly important for LET, since it allows us to understand the level of people’s spoken language ability in both Japanese and English in an objective way.

→ All for One: Creating a common understanding between learners & trainers

Another key benefit of developing our own test is that we have access to all of the data. This is extremely useful as we now have the ability to refer back to any test that we have done since we started in 2018.

Many tests feel like a black box: test takers have no way of understanding why they got the score that they did. And even worse, they have no way of knowing what areas they should focus on to improve. For the MECT/MJCT on the other hand, we have a detailed breakdown of the test takers’ vocabulary, grammar, and spoken communication output. This allows the learner and the trainer to understand what areas should be focused on in a lot of detail.

In addition, LET is able to accurately track the progress of improvement of people in our training programs. At the same time, we can verify the effectiveness of specific programs, which is important because we use external vendors to deliver the majority of the training at scale.

Having the data transparently accessible to learners as well as LET allows us to do things that would be impossible if we were reliant on an external test.

→ Be a Pro: Continuously evolving to achieve subjective-free scoring

It was a very challenging project, to create tests from scratch. But when we looked at the alternatives available at the time, as well as the benefits of having our own test, it was clear we needed to Go Bold and create our own solution.

It wasn’t a perfect solution at the start, and it has been constantly evolving and improving. The tests were originally done as an in-person interview that were scored by LET members based on the CEFR criteria. This method had some subjective elements and was clearly not scalable.

In order to deal with the subjectivity issue, former-LET member @David was the key person that codified both grammar and speaking criteria for the MECT to improve the consistency of scoring. In addition, he made sure all the internal formulas and vocabulary scoring system worked automatically. All of this was possible thanks to his language education background, knowledge of how to apply the CEFR, and spreadsheet skills, allowing for a streamlining of the procedures and a huge improvement in accuracy of scoring.

To solve the scalability issue, former-LET member @rileymasu found ways to automate all of the logistics for both the MECT and MJCT. Continuously optimizing all of the processes, from applying for the test to receiving the results, has been a game-changer in terms of LET’s resource management. In addition, he built dashboards that make it possible for the company to make use of this data in ways that were previously impossible.

Looking at the MECT now, we have removed nearly all of the subjective elements of the test and for scalability, it’s now conducted online with videos as question prompts as well as scored nearly 100% by external scorers. In addition, during this quarter we introduced a new mechanism for scoring vocabulary that is even more accurate and has a wider coverage.

None of these improvements were easy, but all of them were important to get us to where we are now. And I’m very proud of all the work the team has done to continue to iterate this test.

But you don’t have to take my word for it, we wanted to share some learner comments as well. We talked to Adler Hsieh, one of the Japanese learners that has taken the MJCT several times to see what he thought of the test.

Adler Hsieh Comments:
“As far as I know, the majority of tests available focus more on the listening & reading ability, whereas the MJCT focuses more on the listening and speaking ability. I think it’s good for learners to actually be able to get their listening and speaking ability tested. Also, the questions are all related to day-to-day conversations so it’s helpful for us to think about how to respond to them in Japanese.”

As Mercari moves into its next phase of global expansion, I believe the tests will help to identify key people inside the company that can directly support these initiatives. At the same time, the tests are a great tool to track the progress of people working to improve their Japanese or English so they can continue to expand their influence in the company.

If you are interested in finding out more about the language programs or want to know more about specific positions currently available, please check here.