Mercari’s Security Engineering Team: Taking on New Challenges and Supporting a Broad Scope of Functions Across a Flat Team Structure

This article is part of our Meet Mercari’s Security & Privacy Team series, where we will be introducing you to the teams that make up the wider Security & Privacy team at Mercari. This week’s article features the Security Engineering team.

The Security Engineering team works across a wide range of domains broken down into four core functions: Platform Security, Corporate IT Security, Chaos Security, and finally Threat Detection Engineering. We’ll be focused on Platform Security in this article.

The mission of this team is to ensure the security of Mercari’s infrastructure including: Mercari’s microservice architecture, Kubernetes clusters, CI/CD pipeline tools, and cloud infrastructure. While these technologies keep the Mercari app up and running and allow us to achieve rapid development at scale, they also involve various security challenges in their implementation. Let’s hear from the three members tackling these challenges and supporting platform security at Mercari (Simon, Matt, and Suezawa) on what they enjoy about working on security engineering at Mercari.
＊Matt was unable to attend the photoshoot because he was chopping wood for the winter.

Flat team structure supporting four core functions

ーFirst of all, please introduce yourselves.

Simon: I’m Simon, manager of the Security Engineering team. I joined the Security team at Mercari in early 2018, but have been working in security since I graduated from college.

Matt: Hi, I’m Matt and I’m a security engineer here working primarily on microservices and cloud infrastructure security. I’m also working on building out our internal red team. I joined Mercari a bit after Simon and have also been in the security field since I graduated school. I enjoy long walks in the forest to go climb rocks, and chopping wood for the cold winters.

Suezawa: Hi, I’m Hiroki Suezawa and I’m a Security Engineer, I mainly work with Matt on platform infrastructure security. I previously worked at a Japanese security vendor before moving to Mercari in 2019! Currently, I work on many projects with other teams and roles, for example, platform security, corporate IT security projects, new subsidiary security requirements, and automated phishing website detection and response.

Simon Giroux（@simon）

ーWhat’s the mission of the Security Engineering Team and what kind of things are you working on?

Simon: As Nikolay mentioned when he introduced the Security team in the previous CISO Office article, the Security Engineering division is actually two teams: Product Security, focusing on the application and business logic layers of security, and Security Engineering itself which is focused on the infrastructure layer. From the outside, security engineering may appear to be a single team with a single focus, but in reality our team is working across a broad set of domains which can be broken down into four core functions—Platform Security, Corporate IT Security, Chaos Security, and last but not least Threat Detection Engineering.

Matt: While it might sound like we have a lot to cover, we have a very flat team structure and collaborate together across multiple functions. Things move at a fast pace at Mercari, and the company has many talented people. We believe having all these functions under us as one team allows us to handle the pace of progress better and ensure we take a holistic approach to security regardless of the project.

Suezawa: Matt and I focus on platform security in particular. At Mercari, all infrastructure is managed as code, and there are many modern tools used in our infrastructure such as CI/CD tools and cloud technologies. We are working on securing fairly modern systems and to accomplish this, we created a vision and roadmap for infrastructure security. We used this roadmap to identify security risks and design a solution together with Mercari’s Platform Team. We focus on automation to secure our infrastructure at scale. For example, I’ve been working on projects like designing the overall security architecture for a new subsidiary, as well as the security design and implementation for Kubernetes and Mercari’s CI/CD pipeline security.

Hiroki “rung” Suezawa（@rung）

ーWhat makes this team unique?

Simon: For me, I am officially the manager, but this doesn’t mean I am the one making all the decisions. We work as a team, trusting each other, and taking decisions together. I believe our collaborative approach is one of the reasons why the team is doing so well. As long as we trust in each other and learn from our mistakes we can make better decisions given the business priorities, security and privacy requirements and enable Mercari’s Go Bold value while ensuring security along the way.

Matt: Our team is very diverse in terms of technical knowledge and skill sets due to the different functions we support within our team. Our approach to security is more of a hands on, bottom up, engineering style approach, which means we need to take complete ownership of projects and lead them to fruition. It’s also very important to work closely with other engineering teams, like the SRE team and the Microservices Platform team to improve security in collaboration, as opposed to making isolated decisions.

Suezawa: I think the great point of our team is the scope of work we get to challenge ourselves with. We are securing Mercari’s infrastructure across a broad and modern range of technologies including microservice architecture, Kubernetes, our CI/CD pipeline and cloud infrastructure. The kind of modern technology we use is both useful and interesting, and allows us to rapidly develop applications and run them at scale, but the use of these technologies can also create a variety of security challenges, and many emerging technologies don’t follow or have good security practices built in. For me, the best point of our team is to take on new security challenges with these new and exciting technologies and tools.

ーSo what made you all decide to join Mercari in the first place?

Simon: We have so many unknown challenges! Mercari is fluid, it is in constant movement. A mistake would be to think that we can stop everything while we are trying to understand it and fix security problems. It’s impossible to ask everyone to wait while we try to catch up. We have to sail along, follow the chaos, accept that we will not be able to see everything, and trust that as a team we will notice what is important. We have so many talented engineers. Creation must flow.

Matt: I really enjoyed building security from the ground up and felt that I reached the limit of what I could do alone as a security engineer at my previous role. Many engineers I worked with in the past had also moved to Mercari and these were people who have been great mentors for me throughout my career and who I wanted to continue working with and learning from.

Suezawa: I wanted to work on using new technologies, like working with containers in a microservices architecture environment! I also wanted to use English at work. Our Security team is very diverse so I have the opportunity to use English a lot at work. I didn’t use English at all until I joined Mercari, but all our team members are very supportive and inclusive, so I could continue to work and improve my communication skills at the same time.

ーMercari sure sounds like a challenging place to secure. What is your approach？

Simon: We are not trying to become a gatekeeper, blocking the product, quite the opposite. Regarding keeping track of new/ongoing projects, we are all trying to keep up with the constant stream of slack messages. But more than anything, work with our engineers and try to enable them. We focus on priorities, while keeping in mind the big picture and long term plans as much as possible. With the support of the Security Strategy team and Jason, as well as other security team members, we are able to focus on what will have the greatest impact for the business and plan for the long term.

Matt: We try to automate and abstract as much security as possible and build secure defaults, so we need to work very closely with other engineering teams as Simon mentioned. With our SRE and microservice teams we’ve worked together on many things including: conducting PoCs, implementing Kubernetes security tools like Falco/Sysdig, and collaborating to build security boilerplates and guard rails as part of our terraform infrastructure. Projects like these require a very bottom up approach, so when we start thinking about a solution or control we want to implement we need to get everyone involved and use things like design docs to collaborate. This allows us to record the decision making process as we can get consensus and feedback from all teams and stakeholders in a more asynchronous way. I think this collaborative process is very important to make sure we are choosing the right security tools for Mercari and that the tools we choose are well received and effectively used by stakeholder teams.

Suezawa: Yes! Our recent approach is to interface with our platform through abstraction. Mercari adopts a microservices architecture. The Mercari Platform Team provides many templates and abstractions for developers to use. Since we have many developers, It’s challenging to enforce security, so by providing an abstraction layer to developers, we can provide and build security at scale.

Recently I’ve been working on CI/CD Security. By providing secure templates as Actions to developers, they can have secure configuration built in, and our platform and security team can easily modify the abstraction. This makes for a very scalable approach! You can also find out about other examples of abstraction we use by reading our engineering blog.

ーMercari had an incident last year where part of the codebase was exposed, how was the security engineering team involved in handling this?

Simon: Last year we had an incident where due to a vulnerability identified in Codecov (an external code coverage tool* that is used within the company), part of our source code was accessed without authorization.

While I am mentioning that Mercari is stopping for no one. Actually, it does, when it’s really important. In March of 2021, when we found out about the incident, our team raised a red flag, and like the Eye of Sauron, the whole organization stopped what they were doing to focus on what was happening.

Matt: When we got notified about the incident our whole team immediately jumped in and notified all the different teams affected. We had a lot of support from engineers and it was a huge collaborative effort across the company to resolve. Everyone worked through Golden Week and if we didn’t have such a strong engineering culture and understanding/ownership of security I don’t think things would have been as easy. Luckily our team had implemented Sourcegraph (a code search tool) prior to the incident which made identifying and handling potential secrets in the codebase easier.

Simon: Within a few hours, All For One, the whole company worked to address the issues. That was the first time I witnessed this in a company. This required us to communicate with a large range of stakeholders and peers. Suezawa-san played a key role in reporting progress.

Suezawa: Right, there was a lot of attention from the top executives, public relation teams and engineering teams toward the security team. The team had to investigate the extent of the impact, plan containment, eradicate the risks while keeping everyone up to date to the status. This was the biggest security incident so far for Mercari. For me, it was a moment when I experienced firsthand the importance of communication and leadership as well as technology. We still feel we need to improve our security, but this year, we got an award for how we handled things. I was glad when I heard it, as we’ve been sharing our lessons learned publically following the incident.

Matt: We had a lot of work to do following the incident, we took on the task of completely revamping our CI/CD, introducing better solutions around secrets management, and also more general automation and safeguards around our infrastructure as code. We had a few in-house solutions for scanning code for secrets being committed to Github and notifying engineers, but we really needed to ramp up preventative measures rather than just detection. That led us to work together with other teams to create git pre commit hooks tools and CI checks as well as starting to look for more cohesive solutions like Github Advanced Security and other Github offerings related to secret scanning/commit prevention. We also migrated our CI/CD from CircleCI to GCP’s Cloud Build and started working on planning for self hosted Github actions runners where we can better control security around ingress and egress traffic.

Keeping up to speed in a multi-context environment is challenging – but rewarding

ーDo you have a representative project for your team?

Matt： We want to take a more global approach to securing our infrastructure across our different businesses and products. To do this we created a longer term infrastructure security roadmap to help us track and guide our efforts as the company grows. Within we’re focusing a lot of our resources on strengthening the security around our development pipeline as we feel that has the most security impact at our current stage. This includes things like working on further securing access to our resources in the cloud, Github, CI/CD infrastructure, microservices and Kubernetes security. We have a lot of different security mechanisms in place in different parts of the pipeline and infrastructure and we need to make sure we cohesively pull everything together.

Simon: The team isn’t handling operational tasks so much. If I had to describe a typical project, I would say that our team members are being involved early in major projects like Mercoin, or fundamental systems like our production infrastructure platform.

ーPlease let us know if you have any results or feedback.

Suezawa: Some of the results of our work are published externally. For example, I presented on Kubernetes security twice at external meetups, and I also presented on CI/CD Security at an external security conference. I know our team members have won awards internally in the past for our work too I have received the All for one award and Matt has received the Go Bold Award. It’s good to see a lot of recognition and understanding of security at the company.

Simon: Many Security Team members have also written cool blog posts about what we have been working on. It’s worth the read!
https://engineering.mercari.com/en/blog/category/security/

ーWhat are some of the challenges you are currently facing?

Simon: Mercari is expanding quickly and many new projects, new businesses, and new ideas have come up over the years of growth. Maintaining the knowledge acquired over the years can be challenging and it’s important to document enough to safeguard that knowledge and diffuse it as much as possible. As the company grows, the number of contexts that we need to follow are exploding. This is why it is so important to have solid teams whoco-operate and work together. We need the help of everyone to keep track of what is going on.

Matt: Not enough time in a day to secure all the things, there are so many projects I want to work on and contribute to for security but I’m only one engineer. Mercari moves quite fast and our infrastructure is constantly evolving so keeping up can also be quite a challenge at times. Luckily we heavily utilize design docs which we can use to reference and review to help us keep up.

Simon: Overall, communication is a key point across all our challenges. It’s important to stay in close contact with our peers, and ensure that expectations are well managed.

Learning new skills and taking on new challenges is key

ーI know you are in the hiring process right now, but what kind of person do you think would be a good fit for this team?

Simon: The unicorn candidate is an agile developer with golang coding experience, a SRE with GCP cloud infrastructure expertise, and a red teamer who also did log analysis and incident response, and is able to translate geek talk into management briefing notes. I think that having some of these skills is important because understanding the production stack makes our work easier. Since we are also involved in incident handling when something happens, this knowledge ends up absolutely useful to support when it matters.

Matt: An ideal candidate would have strong security and computer science foundations as well have worked as an SRE or done some backend development. We work very closely with engineers so being able to work with and build security around the same tooling is important for our team. Mercari’s environment is quite diverse in terms of talent and techstack, and the environment is quite fast paced so being open to learning new skills and taking on new challenges is a must.

Suezawa: Yeah, and I’d like to work with members who enjoy engineering and enjoy working with the product side.

ーWhat opportunities do you see in Mercari’s current phase?

Simon: Mercari wants to expand globally. This means that we will need a wide range of experience and a diverse cultural representation. In the context of our team, we are seeing each company requiring different types of infrastructures. Different infra and methods means an increase in complexity, which makes it difficult to maintain. This also means that we have an infinite source of cool projects to work on, with a challenging balance between speed and security alignment.

Suezawa: Members can have ownership of projects they work on, and members can work on new security challenges such as securing cloud environments, Kubernetes, platform infrastructure, and supply-chain security. I worked at a security vendor before, back then, I couldn’t imagine real security concerns customer facing product companies have. But at Mercari, you can work on unique and interesting security challenges. I believe It’s a great opportunity for many security engineers.

Matt: There’s still lots for us to improve and reiterate on as we build and maintain a security roadmap for our infrastructure as we move towards a more global approach. There are various improvements we want to make with automation and more areas we need to build stronger controls around in our supply chain.

ーWhat are your goals for the future?

Simon: What I hope is to see our members act as staff engineers, joining major projects as dedicated members, answering any security needs the project might have and leading security architecture decisions from the start. We are already supporting projects and new companies, but I think we can deepen our involvement and impact on Mercari’s endeavors.

Matt: We are also hoping to have more resources dedicated to specific areas in platform security to work more closely with our engineering teams, as well as building out our internal red teaming efforts.

Suezawa: I want to make the security team even more global and have our security practices competing on the global stage with larger tech companies.

ーOne final message for our readers!

Simon: Like I am saying to many candidates I am meeting during interviews, Mercari is a challenging place to work at. It’s quite chaotic. It goes in all directions all the time. At the same moment, work is fun and what we are working towards is exhilarating. I feel like we are surrounded by skilled and inspiring people within as well as outside of the team. While we rarely have time to be bored, we try to reserve time to unplug, play boardgames after work and our team is more than anything just a group of great friends.

Suezawa: You can grow as an individual contributor and you can take on many challenges at Mercari! Mercari has given me the opportunity to take on many new experiences with exciting and new technologies and business areas and if that sounds interesting to you why not join us!

Matt: If you are interested in security architecture and chaos engineering please apply! Mercari is a fun place for engineers and provides a lot of opportunities for career growth.