Senior Machine Learning Engineer (LLMs & Prompt Engineering)

Global Market for QA

Markets are changing rapidly, and there is a demand for quick software releases. Agile development has become mainstream; on a global scale, 92% of developer teams adopt agile development, 71% of which aim to release at least once a week. With such a rapid cycle, manual verification is too time-consuming, and there is an urgent need for automation. Our services can solve these critical challenges and make our clients a competitive advantage in software development. QA’s global budget is said to reach over $1 trillion. We are rapidly expanding our service reach internationally to win this vast global market.

About Us

Autify, Inc. is a San Francisco-based startup that was founded by the first Japanese team to graduate from Alchemist Accelerator, one of the top accelerators in the US. With the mission to empower people’s creativity through technology, we have been offering an AI-powered automation platform aiming to eliminate the test phase.

Since 2019, we’ve been providing our AI-powered nocode test automation product and have grown the team to 100+. Our next move is to double down on GenAI. As we have been investing in AI and helping customers’ software quality since 2019, we know how to best utilize GenAI in software QA. With that, we have recently launched our new product, Genesis, a GenAI agent for software quality assurance. 

About This Role

Job Purpose

You will mainly be responsible for developing a new product utilizing machine learning techniques and generative AI from a proof of concept into a releasable product.

Responsibilities

• Create effective prompts to maximize the quality of the output from LLMs based on the product specifications
• Build our machine learning models using data relevant to the software testing if needed
• Work with other engineers, designers, and product leads to implement various features while considering performance, security, and reliability
• Create PoC Web/CLI applications that use LLMs or ML under the hood to validate the feasibility of ideas
• Proactively identify functions that can be improved, and suggest possible solutions in the technical aspects and product-related ideas
• Participate in on-call rotations for incident & alert handling

Required Qualifications

• Being able to work in Japan standard time (Working partially in JST can be discussed)
• Over two years of experience in training and customizing deep learning models, especially LLMs
• Over a year of experience in prompting open-source and closed-source LLMs.
• Experience in developing APIs with FastAPI and Python
• Up to date with the latest Machine Learning Research especially on LLMs and Prompt Engineering 
• Experience in building metrics for evaluating various tasks LLMs. 
• Ability to communicate and work in English

Nice-to-Have

• Knowledge of natural language processing and visual recognition
• Proficient in the process of analyzing, hypothesizing, and testing
• Experience in MLOps
• Modern development experience with Agile or Scrum
• Experience in E2E test automation with Selenium and Appium

Selection Process

1. Hiring manager interview: 45 mins
2. Technical interview: 60 mins
3. CEO interview: 60 mins
4. HR interview: 45 mins

About the Engineering Team

Our Team

Autify's engineering team is international, with more than half of the engineers being foreign nationals and all developers being English speakers. We have a good culture where members respect each other's backgrounds. Since we have two products, Autify for Web and Autify for Mobile, our development engineers are divided into two teams corresponding to each product.

We have about 30 front-end, back-end, and technical support engineers who work closely with engineering managers, product managers, QA managers, designers, and machine learning engineers.

The product manager is in charge of the general roadmap. The product manager develops a rough roadmap and works with the engineers and designers on the details of each feature and bug fixes before implementation. Depending on the size of the feature, we appoint a feature lead from among the engineers to be in charge of the project and give them the discretion to make technical decisions.

Our Development Process

We create a ticket about what kind of function we will develop. In some cases, the product owner creates it based on user feedback; in other cases, each team creates bugs and investigation requests directly. 

The development cycle consists of one Sprint, which takes two weeks. Development tasks are assigned during the planning stage on the first day of the Sprint, and we hold a daily standup meeting to check the overall progress and any problems. (If there is a time difference, it is done at lunchtime).

We review code on GitHub. After the review, the merged branches are automatically deployed to the staging environment, and E2E tests are performed (which Autify does itself).

After testing on the integrated branch, it's deployed to the production environment. Deployment is done several times a week as needed. We also hold sprint ceremonies like Sprint planning, feature refinement and sprint retrospective sessions. The developer will demonstrate the developed function to all employees at the end of the Sprint in the Sprint Demo.

Tech Stack

• Backend: Ruby, Ruby on Rails, Go
• Frontend: JavaScript, TypeScript, React, Redux, Styled Components
• Machine learning: Python, PyTorch, Kubernetes, GPT-4
• Database: PostgreSQL, Redis
• Test: Appium, Selenium, WebdriverIO, Puppeteer
• Infrastructure: AWS, Docker, Terraform, Packer, MacStadium, GoogleCloud
• Monitoring: Datadog, New Relic, Sentry
• CI/CD: CircleCI, GitHub Actions
• Source code management: GitHub
• Project management: Jira
• Misc.: G Suite, Slack, Notion, Gather, Zoom, Miro, 1Password, Mixmax, Calendly, etc.

Employment Conditions

Employment Status

• Full-time contractor

Work Location

• Remote work in JST timezone (partial JST overlap negotiable)