The Product Security Team: Supporting All Phases of Mercari’s Product Lifecycle

This article is part of our new “Meet Mercari’s Security & Privacy Team” series. In this series, we will introduce the sub-teams that make up the Security & Privacy Team at Mercari. This month’s article features the Product Security Team, which helps developers design and develop secure products from the requirements phase through release and maintenance.

The team members come from various countries and backgrounds, and work on a variety of projects with a flexible structure.

Meeting both the latest security standards and users’ privacy and security expectations

──What is the mission of the Product Security Team?

@nikolay: The mission of the Product Security team is to ensure that all of Mercari’s products and services meet both the latest security standards and our users’ privacy and security expectations. In addition, we do our best to align with Mercari’s business goals and support our product teams in shipping secure products.

Nikolay Elenkov (@nikolay)

@Azeem: We want to enable and assist developers to design and develop secure products from the requirements stage all the way to release and maintenance. Every product we ship should be designed with the security and privacy of our users in mind, without placing a burden on development teams and instead enabling them to move fast and ship high-quality products.

──What makes this team unique?

@Azeem: The team has members from several different countries: Bulgaria, China, Hungary, Thailand, UK, and USA. It’s diverse and accepting. There is a big opportunity for members to work on and research things that are interesting to them. For example, we once worked on our own secret scanning solution as well as our own vulnerable web application.

@Eli: I would say one great and unique culture here is the good relationship between us and developers. Our team is always trying to cooperate with the development team instead of acting as a doorkeeper. We try to actively get involved in the early design phase to have threat modeling and security design review together with the development team, and developers from this company also care about security so they also reach out to us for potential bad designs, system risks, etc.

──So what made you all decide to join Mercari in the first place?

@nikolay: I was initially interested in Mercari’s modern development and deployment stack (Go and microservices architecture). After joining, I found out there is much more going on behind the scenes, with developer, product, and security teams constantly working to stay up-to-date with current technology and make the most of Google Cloud. I was fortunate to join a very driven and welcoming team, and ended up liking the culture as well.

@Azeem: The company has a really great working culture. Social life in Mercari is really great. There is no such thing as “Social Atomization” in Mercari because you’re always encouraged to join team lunches, team building events, and company clubs!

Azeem Ilyas (@Azeem)

@าirat: I love the culture here. It is hard to describe correctly, but the balance of interpersonal relationships and professional relationships with your colleagues here is invaluable. Fun fact: our JP region’s CEO, Naoki-san, just walked up to Azeem and asked if he could get a Security Champion T-shirt the other day.

Naoki Aoyagi, Senior Vice President of Japan Region / Chief Executive Officer of Mercoin, Inc. and honorary Security Champion!

@Eli: One thing I would like to add is that working in Mercari really provides more chances for self learning and growth. We can always propose and try new solutions/tools to see if they can help improve our daily work. At the same time, there are also many training/conference opportunities for us to expand our skill sets and meet more talents from outside.

@viktor: Mercari’s culture was the main pull for me. A friend of mine already worked here, and she would talk about how great her colleagues are, and all the activities they do together. It feels like Mercari really understands that work is a social activity, and that team cohesion and motivation are just as important for productivity as individual skill. Because of this, there is a really great atmosphere in the office, and working becomes an enjoyable collaborative effort.

Mercari from the outside seems like a simple company with a very practical product and business model. Only after joining did I realize the depth and width of the business activities Mercari conducts. It was also very interesting to see how committed Mercari is to good communication across language barriers, and in general. Perhaps the most surprising thing for me, was that the most important things did not change. Many times companies seem very inviting and pleasant to work at from the outside, but Mercari is also very nice from the inside.

Viktor Ferter (@viktor)

The importance of security is well-understood among developers, PMs, and management

──Can you share some of the outcomes or feedback for your projects?

@nikolay: As the Product Security Team, we are always working to support the latest products and services Mercari is developing. We may work closely with developer teams on purely engineering problems, like migrating our Web application to microservices, security access tokens in Web apps, and promoting DevSecOps. Sometimes we are closely involved in product design or UX discussions, or may be helping out with the PCI DSS certification of our payment services.

Our internal projects also aim to support and lighten the burden on our developers, while maintaining security, as Azeem mentioned. We are always working to improve automation and optimize our Secure Development Lifecycle (SDLC). You can find more details in the articles we have published on code scanning and mobile security. We research new technologies, assess how to best adapt and integrate them into Mercari’s development process, or even get back to basics and implement encryption algorithms from scratch.

@execjosh: I mostly work on reviewing design docs and improving the code quality of our internal security tools. I also work closely with the Threat Detection Engineering team on keeping our SOAR system in line with the latest Go design patterns and best practices.

Joshua Williams(@execjosh)

@Azeem: There can only be so many Product Security engineers at one time, and with many different subsidiaries, it’s impossible for the security team to be everywhere all the time. For that reason, we started the Security Champion program at Mercari. The program, which is run periodically (in cycles), encourages developers to become Security Champions, teaching them important security principles and diving into specialized security topics and labs. At the end of each program cycle, we grant rewards to our champions based on their contribution. These champions often go on to find and fix major security issues in their domains, from both a design perspective and at the code and dependency level.

@Eli: We have successfully integrated GitHub Advanced Security into key GitHub repositories. By doing this, we could automate static code scans and secret scans for each target repository. From the results, we can further track down issues and fix problems earlier. There are also many other great features such as push protections to avoid including any credentials in the pull requests, and a security board summary for each repository. We are now trying to create some more security badges to display on each repository to represent the health status.

@าirat:My main focus is on security design review, consultation, and penetration testing of various products and services. Recently, I have also become a PCI DSS Internal Security Assessor (ISA) due to my heavy involvement with Merpay. We have been relying on external auditors for a while, and we want to change that. I am now at the start line to automate the audit as much as possible to reduce the burden on people involved in the process.

@nikolay: As most things in software and security, our projects are constantly evolving. We continuously collect feedback from developers, and strive to improve and optimize our solutions to minimize the burden on the development process, reduce false positives, and generally improve usability. Sometimes that’s as simple as changing a setting of a third-party tool; sometimes it’s a fairly long development process that requires planning and coordination with multiple stakeholders. One example was adapting Github’s dependabot to play nicely with our multiple internal Go projects that are widely used across microservices. That involved designing a temporary token granting system and a method to seamlessly integrate with all projects that use dependabot. Hopefully we can share more details in a future post.

@Azeem: For each Security Champion event we hosted, we actually collected feedback and a score out of 10 from each attendee. Thus, we were actually able to determine an NPS score for each event. Most of our NPS scores fell in the 60–80 range, which generally means a good rating according to guidelines on NPS scores. We also conducted round tables with our participants and found out that, in general, they wanted more practicals and less theory. This and other feedback we received helps us dramatically improve for our next cycle of Security Champion training.

@าirat: I would say the fact that the importance of security has been well-accepted among developers, PMs, and management level is the best outcome I could think of, because it makes other projects that everyone is working on run smoothly.

Common to all members is their curiosity and ability to work at different levels of abstraction

──What challenges does your team currently face?

@Azeem: The company moves very quickly! Rapid adoption of new technologies/domains means that the Product Security Team needs to know how those technologies/domains work, their design limitations, and security implications. While it might not be great for someone who prefers a stable environment, it’s great for someone who likes a chaotic fast-moving development process and loves to learn and break new technologies.

@nikolay:In addition to keeping up with the latest technologies and products, we need to quickly adapt to the team and structural changes inside the company. Mercari does not mandate a single project management or development process, so we need to be equally flexible when working with teams across the company. When it comes to tools and technologies, Mercari’s culture is predominantly bottom-up, so if you want to ensure wide adoption of a security tool or process, you need to both convince engineers that it’s a good idea and prove that it will not hinder development speed.

──What type of person do you think would be a good fit for the team?

@Azeem: Definitely people who like to break things apart. The kind of people who are not scared to jump into a large system like Mercari and figure out its in and outs quickly, then identify gaps which others may have overlooked. Usually, developers are the best people to turn into Security Engineers, but we’ve even had people join from non-tech backgrounds like language interpretation. It really depends on how driven an individual is and how quickly they can grasp the workings of a system.

@าirat: If you are a developer who has been doing security as a hobby, more than half of the team, including myself, are of this type. Like Azeem mentioned, the system is big so there are many things to learn, relearn, and double-confirm here. Things you think you know well could be drastically changed in less than a month. If you like this kind of endlessly changing environment, we are open for a chat even before applying. We also love people who can come up with many alternatives for short/mid/long term solutions, then discuss possible outcomes or middle grounds among stakeholders.

Nirat Tatiyanupanwong (@าirat)

@nikolay:As Azeem and Nirat mentioned, we welcome people from different backgrounds. Curiosity, being able to work at different levels of abstraction, and broad knowledge of computing are some of the common things among all our members. There is no one single path to becoming a security engineer, and it is never too late to switch careers if you are up for the challenge. Maybe we can get into more detail in a future post, but I feel that these two articles by Parisa Tabriz and Chris Palmer are a good summary of the different paths to becoming a security engineer and some of the challenges along the way.

Being open-minded and flexible without compromising on security are quite important for being a successful security engineer at Mercari. People who are used to mandating rules and pushing out security solutions without getting feedback from the people who will be impacted by those rules and solutions generally do not do well at Mercari.

Additionally, while product security is very much about communication and design, solid engineering and software development foundations are extremely important. While automated scanning tools improve every day, they are only a small part of a mature security process. People who come from a tool-driven penetration testing or check list-based compliance background usually have a lot of catching up to do at Mercari.

──What opportunities are available to members of the Product Security Team at this stage?

@nikolay: As we mentioned before, Mercari engineering uses a modern development stack and process and is always keeping up with the latest cloud developments. So if you are interested in securing cloud technologies, working on the automated vulnerability detection rules, and collaborating with development teams that are pushing the limits of mobile and Web development, Mercari is definitely the right place to be.

As a product security engineer, you have the freedom to use the tools you like (both open source and commercial), build your own tools and release them to the security community, and attend online or in-person trainings and conferences to keep up with the latest developments in security. You also have the opportunity to impact product and service development from their inception phase.

As a company, Mercari has a fairly flat organizational structure, and our culture of trust and openness makes it easy to approach people, get access to information, and not just question design decisions, but have a voice in product and company direction as well.

Mercari’s engineering organization is also very diverse, so being at Mercari affords you the opportunity to work not only with different technologies, but with people from very diverse countries and cultures as well.

@Azeem: We have lots of opportunities to experiment and work on new technologies and projects. If you have an idea, it’s not a problem to propose the idea and be given lots of time to make it work.

@Eli: The company has provided many online trainings for self learning. Apart from that, employees can also set up their personal OKRs (objectives and key results) to seek new challenges on tech exams and certificates.

Shaokang Sun (@Eli)

@าirat: Our team is small, so we could use extra hands and eyes on everything we are working on or things we have not yet done. As long as you have the ability to work with the team and bring your own ideas into projects, then you’ll have unlimited opportunities (except for a budget that won’t be unlimited LOL).

Aiming to support Mercari’s product development and stay on the cutting edge of security

──What are your team’s goals for the future?

@Azeem: To be able to build our own tools without relying too much on vendors (and keep them maintained!). Also, to open-source world-class tools that we can share and maintain with others on a global scale.

@nikolay: We want to continue supporting product development at Mercari, while keeping up with the latest trends in software development and security. As Mercari diversifies its business and goes into new domains, the Product Security Team will need to scale accordingly and get experience in each domain. Securing fast-moving fields such as IoT or AI-based technologies is always a challenge, but there is also ample opportunity for innovation and research.

@viktor:Since Mercari uses cutting-edge technology, there is a real opportunity to identify novel attack vectors, and to create security tools that are able to prevent them. Researching and developing in this field will improve the security of Mercari and eventually, the security of the internet in general.

────One final message for our readers!

@Azeem: We have some projects that we’d definitely love some help with. And members are always free to propose their own projects, too.

– Automated DNS discovery of new applications and onboarding them to DAST tools
– Home-grown SCA and Supply Chain Analysis tools
– Automated repository and service risk assessment
– Development of an internal CTF platform with challenges
– Developing and improving our inhouse education program using a gamified experience

@nikolay: As we work towards the shared vision of Mercari security (‘Security and privacy by design, by default, and at scale’), we welcome people who want to get involved in all aspects of security, support products throughout their lifecycle, and feel challenged by implementing security at scale. If you like a fast-paced working style, cloud technologies, automation, and securing mobile and Web applications, please consider joining us!