https://canva.link/xkcqxjkagx14fty
https://github.com/nevenjer/companiesmarketcap-playwright/tree/main
I combined knowledge from multiple online courses in Software Testing, Test Automation, Programming, Data Engineering, SQL, and Data Analysis to design and build this project.
I also used AI creatively as a development and learning partner — to explore ideas, analyze problems, improve implementation approaches, debug code, and validate solutions.
Rather than relying on a single technology or course, I applied knowledge across different areas to design a more structured, efficient, reusable, and scalable solution.
Automated data collection, validation, and storage pipeline using Playwright & TypeScript.
This project aims to create a reusable company symbol dataset sourced from CompaniesMarketCap, enabling the use of a large volume of company symbols across market data systems, R, and SQL Server.
https://github.com/nevenjer/companiesmarketcap-playwright/blob/main/docs/development-notes.md
→ Market Cap validation → DOM inspection → First company → 100 rows → Advertisement rows → data-sort → Pagination → Privacy / browser interference → Page Object Model → Scraper Service → Data Model → JSON Storage → Upsert → Duplicate Prevention → Ranking History → Metadata → Validation → Idempotency → Final Dataset → Lessons Learned → Future Development