Mercari’s Threat Detection Engineering Team: Continuously improving to provide autonomous cyber threat detection and response at scale

This article is part of our new “Meet Mercari’s Security & Privacy Team” series. In this series, we will introduce the sub-teams that make up the Security & Privacy Team at Mercari. This week’s article features the Threat Detection Engineering Team.

The Threat Detection Engineering team, as its name suggests, was established as a group focused on detecting threats targeting the company. This team consists of members who worked on the SOAR project, a platform for the detection and response of cyber security threats and attacks. While still a relatively new team, they have a great opportunity to build a foundation that will support the company for years to come. This is no easy task, and in order for the team to be successful, they will need to cover all of Mercari’s systems and services. We spoke with members of the team (David, Anna, Max, and Muge) on how they tackle the highly complex task of detecting malicious or unusual behavior and identifying cyber threats.

Detecting malicious and abnormal user behavior to identify potential cyber threats

ーBefore we get started, could I have you all please introduce yourselves?

David: My name is David, and I’m the manager of the Threat Detection Engineering Team. I first joined the security engineering team at Mercari in early 2019, and have been working primarily on security monitoring, incident response and securing our cloud infrastructure. Back in 2020, I started working on a side project to automate our security alerts, while it began as a side-project, over time this ended up evolving into a fully fledged SOAR (security orchestration automation and response) platform which we use for detection and response across various systems at Mercari [EN/JA]. To further support the development of our SOAR platform, and bolster our cyber threat detection capabilities we decided to create a dedicated team that could focus on these efforts which I am currently leading.

Anna: My name is Anna, I am part of the Threat Detection Engineering team. I joined Mercari in December of 2021, initially as an overseas subcontractor from Hungary, and finally moved to Japan this year in March, officially becoming a full time employee in April. In addition to my work on threat detection engineering, I am also working on chaos security as part of our internal red teaming activities.

Max: Hi, my name is Max. I joined Mercari at the same time as Anna, but from Austria instead of Hungary. Since our timeline and teams are exactly the same I am going to keep this short, but I am basically working on the same things as Anna.

Muge: I’m Muge, I recently joined the Threat Detection Engineering Team, so I’m still being onboarded and learning from the others, but I’m excited to be part of the team and work on threat detection engineering at Mercari!

ーWhat is the mission of the Threat Detection Engineering Team? What kind of stuff are you working on?

David: As I mentioned briefly in my introduction, the Threat Detection Engineering team was established very recently. But, all our team members were originally members of the Security Engineering Team working on the SOAR project—a platform built to detect and respond to cyber security threats and attacks—. To better accomplish our mission we decided to carve out a team entirely dedicated to threat detection. The goal of threat detection is to identify cyber threats by analyzing our environment with specialized tools using event correlation, threat indicators, and user behavior to detect malicious or abnormal activities.

Max: In contrast with the security engineer role, which builds out secure infrastructure and security controls to prevent breaches, a threat detection engineer starts from the assumption that there is a breach and works to identify and handle it. This is a very complex task, and we need to use all the tools available to us to detect malicious activity.

Anna: Other than tools, we also have to maintain a deep understanding of (not to mention keep up to date with) attacker techniques and their behaviors. We need to design detection systems and rules so that we’re able to detect attackers as early as possible, this way we can start remediation efforts as soon as possible.

Muge: Having a team focused on limiting attacker dwell time is very important, many global reports and statistics show that, without persistent monitoring, advanced persistent threat actors (APT) can often stay undetected for as long as half a year. This can really have a huge impact on a company and this is why we are focused so much on early detection.

Left to Right：David Chapdelaine, Maximilian Frank, Anna Simon, Muge Cephe Dursun

ーWhat makes this team unique?

David: I think what makes the team unique is the diversity of roles that each threat detection engineer has to cover. For example, we develop and design our own security services and automation, therefore everyone works as a software engineer, we also manage and maintain all the infrastructure and deployment pipelines, which means we also work as an SRE. But that’s just the beginning and there are many more hats to wear!

Max: To be successful, our team requires visibility on all systems and services at Mercari. To achieve this, we have to work across many different teams, departments, and business, as well as on multiple projects. Collaborating across diverse and multi-functional teams is both one of the most exciting and challenging parts of the job.

Anna: When a security incident occurs our team becomes the bridge between the CSIRT (Computer Security Incident Response Team) and other security teams and engineers. It’s a very unique position at Mercari, and I enjoy that it allows me to work with a lot of different people.

Muge: I feel that everyone on the team has an engineering mindset in that we’re able to build our own tools, but with a security focus— this is what excites me about working here.

Max: Yes I agree, compared to the security industry in general, we look at detection from an engineering standpoint, so we don’t have just a classic SOC (Security Operations Center) with tiered analysts, our involvement is broader and everyone is involved in the improvement of alerts and automation. A side effect of this is that due to the broad skillset we are looking for it can make it harder to find people with the right skills to join the team, but it’s a more rewarding path I think and good for career growth. For anyone working in a SOC looking to expand out into a more engineering focused role – I would highly recommend taking this kind of approach.

Anna: At my previous job, we were also scripting and building our own tools. However, at Mercari our team focuses on modern software practices and technologies, like running on the cloud and continuous deployment. For example, our detection rules are all written as code, many in Rego with Open Policy Agent (OPA)—doing things this way also allows us to stay up to date with the latest technologies also used by other Mercari engineers.

Muge: There’s also the fact that a detection engineer at Mercari has to accomplish so many unique roles, like threat intelligence analyst, cloud forensics expert, threat hunter, red teamer, incident handler and so on. Working on so many different security domains can be challenging but there is a lot to learn from doing so.

ーSo what made you all decide to join Mercari in the first place?

David: I’ve been working in the security industry for many years, from offense to defense, and when cloud providers and cloud-based technologies started to emerge I thought it was really going to transform the security industry. At that time, I decided to transition to a role in devops and cloud engineering, which was really fascinating, but I wanted to go back to a security role with a focus on securing a modem architecture running on the cloud. I think Mercari was the perfect place to do this and Mercari’s heavy use of Kubernetes and modern microservices architecture was very attractive to me. Also, I was seeing Mercari engineers speak at lots of external conferences where they would openly discuss their engineering processes. The culture and values of the company were also key factors in my decision to join Mercari.

Anna: While I enjoy studying Japanese I wasn’t really comfortable using Japanese in a business setting yet. The possibility of being able to use English at a Japanese company was really interesting to me. At the same time, I also get to work using Japanese and I’m able to grow my language skills too.

Max: For me it was the chance to have a flexible working environment. I read a lot of good things about the culture at Mercari before applying and could really feel Mercari was a good fit for me during the interview process. Also, work here is not all about work— there are many various opportunities for me to enjoy spending time and relaxing with my coworkers as well.

Muge: As for me, I was looking to get more experience working in an engineering environment and Mercari is definitely a good place for that. Also, Mercari’s values really resonated with my own personal values.

Focus on expanding detection capabilities and coverage to meet security requirements

ーWhat kind of projects does your team work on?

David: Everyone in the Security & Privacy Team is focused on protecting and preventing security breaches, we put a lot of effort into these endeavors. However we can’t prevent everything, this is why we need to detect breaches as soon as possible and remediate them as quickly as we can. To that end, our projects are either ones that we initiate to improve our detection and response capabilities, ones that are spearheaded by Mercari’s various departments and businesses where our team’s focus is on extending our detection capabilities and coverage and making sure requirements are met.

Max: For the SOAR platform that we’re developing, we need to integrate various threat intelligence and IOC (Indicator Of Compromise) data feeds. A project I’m currently working on is to build a system that can ingest those types of data and allow us and the CSIRT team to visualize the data in an efficient way. At the same time we’ll also use the data to improve our detection rules and enrich security alerts.

Anna: When I started working on our SOAR platform, an issue that the team was experiencing was a lack of visibility on the health of the services we were running, for example we could have an outage on a collector, but it would take too long for the team to notice it. A project I led was to add monitoring on top of our security monitoring platform using Google Cloud Monitoring. For this, I had to expand a Golang package called zapdriver, which I open-sourced, and also built various dashboards and metrics “as code” using Terraform.

David: It’s also typical to work on other company’s projects—right now, we’re really busy with the launch of Mercoin. Our team works closely with stakeholders to implement and meet all the necessary security requirements in relation to security monitoring and security log retention.

ーWhat challenges does your team currently face?

David: As a new team, we have so many exciting challenges in front of us. First though, we have to define and structure how we will work as a team to deliver high value to Mercari and also build strong relationships with the key stakeholders of the business.

Max: We have lots of different types of logs to ingest, sometimes they have no documentation for their schema, or field types fluctuating, which makes the jobs harder. The amount of data to ingest has a related cost, so it’s important to collect and filter events that are of relevance for security or for the purpose of finding anomalies and attacks.

Anna: Log coverage is definitely a challenge, but an even bigger challenge is to build quality detection rules using the data we collect. We have to think of all the possible attack vectors and how we can detect them. We keep revising our rules and adding new ones as new techniques are discovered.

David: Yes, it’s very important to stay up to date with the latest tactics, techniques and procedures used by threat actors. The challenging part is balancing our time between research, engineering, and response.

Muge: The knowledge domains and technologies involved are also very vast, this adds to the challenge of identifying threats. One day you might be working on a new malware impacting Windows endpoints, and the next you’re analyzing a potential security defense circumvention in a cloud deployment.

Max: The attack surface is always expanding, but by engineering our tools and detections for scale, we’re able to tackle that challenge head on.

We have the opportunity to build the foundation of Mercari’s security for many years to come

ーWhat kind of person do you think would be a good fit for the team?

David: A person with very good knowledge on cyber attack tactics, techniques, and procedures, as well as the know-how to transfer that knowledge into operationalized detection would be a good fit. Mercari keeps growing, so it’s important to also be able to automate a large portion of our tasks. For that reason, someone able to develop and maintain tools is essential.

Anna: Someone who is eager to learn new things, knows programming, and is comfortable developing their own tools to solve issues.

Max: A person who wants to grow their career in security. Mercari and our team is a good place for growing as a professional. It’s possible to take initiative and start and work on your own projects, so someone highly motivated and self-driven would be perfect.

Muge: Technical and analytical ability are probably the most important skills to have as you’ll need them to work through security investigations.

ーSo what kind of person is not suited for the job?

David: The security threat landscape is constantly evolving, and adversaries are always a step ahead. We have to constantly adapt and move quickly, sometimes even puting aside our daily tasks to address a new vulnerability or an investigation. This can become, at times, very chaotic. Someone who would have difficulty navigating through this may not be the best fit for our team.

Anna: During incident response, we have to work quickly and keep up with lots of new information. If someone can’t work under pressure, they wouldn’t suit the role.

Muge: Someone who likes to have repetitive tasks or have all of their tasks perfectly defined wouldn’t enjoy working on our team as we need to be very creative and come up with novel techniques to detect malicious activity.

Max: Everyone has their own tasks, so everyone is able to be autonomous, but it is also vital for all of us to be able to work as a team. Therefore, this team may not be suitable for people who like to work independently and solo all their projects. We all contribute as a team to the SOAR platform and support each other in building the best platform we can through collaboration.

ーWhat opportunities are available to members of the Threat Detection Engineering Team at this stage?

David: As a new team, we have great opportunities to help shape the direction of the team and build the foundation of security at Mercari for many years to come. We have to expand and develop our core practices to many other domains, like threat hunting, threat intelligence, and security analytics. There is still a lot of room to grow.

Anna: You can expand your career in many directions, there is always something new to learn. For example, I’m currently studying web application exploitation, as well as preparing for a related certification.

Max: With the “Your Choice” work from anywhere in Japan system, and full flex time, you have the opportunity to manage your own time to fit your daily life, like taking care of your family or your side projects. If you like to travel, you also have the possibility to work temporarily from anywhere in Japan, enjoying a “workcation” lifestyle.

Muge: I like being part of a company that keeps growing, it provides many career opportunities and I can continue learning new things.

Pioneering a unique approach to improve Mercari’s detection and response capabilities

ーWhat are your goals for the future?

David: The future of our team is, through continuous improvements (“kaizen” in Japanese), to continue delivering our mission with the best technologies possible and provide an autonomous cyber threat detection and response at scale.

We see threat detection as a continuation of the shift left strategy from the SSDLC (Secure Software Development Lifecycle) that we initiated at Mercari. We want to keep pushing the detection of security breaches towards the very beginning of the adversary attack flow, using processes from the domains of software development, delivery, and system reliability engineering.

The discipline of detection engineering is quite new. Therefore, there are currently no consistent methods, standards, or frameworks to base our direction on. Our ambition for the future is to pioneer our own approach, and continue sharing with the community, while incorporating these concepts to improve Mercari’s detection and response capabilities.

ーOne final message for our readers!

David: With everything we discussed today, I think if someone is looking for this kind of experience and challenge, Mercari is definitely the right place! We have a wide variety of projects so there is also something new to try. Ultimately, our goal is to provide a safe and secure (“anshin, anzen” in Japanese) experience to our customers, but we can also have great fun while accomplishing it too!