@maaretp our automated tests run on 39 azure agents with 4-22 pabot processes on each. Test duration including Build & IaC deployments take still about 90 minutes. Hindrances of monolithic architecture hit hard since we can't limit the builds, IaC and tests to the changed parts of the software.
We have over 5000 automated test suites.
We get around 8 PRs merged daily.
@borisv we got excellent results from making automation tests atomic (incl. users per suite) and merging test reports separately.
Our current test automation runs below 30min using 17 test automation runners executing minimum 2 suites parallel ( some API tests run 8 ) executing 3 675 tests on Azure pipeline agents. Total test execution time is 5:55:13.
We use #RobotFramework & #Pabot
Spent a day battling #Azure SQL Server transaction deadlocks occurring in sharded and parallel #RobotFramework testing using #Pabot.
Was fun still I expect it's because:
- #pyodbc isn't very efficient deleting rows from multiple tables in one transaction.
- ODBC driver isn't really the latest and greatest so maybe updating it might have some positive effect.
Matti Järvinen