Using Gen AI for Test Ideas… an experiment!

16 min readMay 8, 2024

This article was first publised on LinkedIn

I have been experimenting with various LLMs and exploring what generative AI can do and what it can’t. One of the most popular use cases for generative AI is the ability to generate test ideas. Yesterday, while scrolling through my LinkedIn feed, I saw a post from Chandan Gupta where he spoke about using chatGPT to generate test cases. He mentioned the example of generating test cases for the login page of an e-commerce application. I thought I might as well give it a try and see what kind of test ideas does chatGPT come up with.

Please be aware that I used chatGPT 4 which is available as a part of the pro plan. My results may differ from the results given by chatGPT 3.5 which is available as a part of the free plan. Remember the trick with any of these models is to make your prompt effective enough to give it enough context for it to help you. So here’s what I did and the results…

Me : I am looking for test ideas for testing the login page of an e-commerce platform. Please ask me all possible questions that help me to come up with the Test ideas

Note that letting chatGPT to ask questions allows it to build context…

ChatGPT: Certainly! To help you develop comprehensive test ideas for the login page of an e-commerce platform, I’ll need to gather more details about the system and your testing objectives. Here are some questions to consider:

1. Functionality and Requirements:

- What are the basic functionalities of the login page? (e.g., username/password entry, remember me, password visibility toggle)

- Are there alternative login methods available (social media login, biometrics)?

- What are the mandatory fields on the login page?

- Are there specific requirements or rules for the username and password (length, special characters)?

2. Security Concerns:

- How is the user authentication handled?

- Are there any password policies in place (minimum length, complexity requirements)?

- What happens after multiple failed login attempts? Is there an account lockout or captcha challenge?

- Is there two-factor authentication, and how does it work?

- How are the login credentials stored and transmitted? Is the data encrypted?

3. User Interface and Accessibility:

- What does the login page look like? Can you describe the layout and elements present?

- Is the login page responsive to different devices and screen sizes?

- How accessible is the login page for users with disabilities (screen reader compatibility, keyboard navigability)?

- Are there any specific UI elements like tooltips, error messages, or help links?

4. Performance Expectations:

- What is the expected load time for the login page under normal conditions?

- How many users are expected to use the login function simultaneously?

- Are there performance benchmarks or targets that the login page needs to meet?

5. Integration and Environment:

- How does the login page integrate with other parts of the system (e.g., shopping cart, user profile)?

- What environments does the login page need to support (web, mobile, app)?

- Are there third-party services involved in the login process (e.g., OAuth providers, security services)?

6. Error Handling and Messages:

- What error messages are displayed for invalid inputs or failed logins?

- How user-friendly and informative are the error messages?

- Are error messages logged somewhere for troubleshooting?

7. Testing Logistics:

- What types of testing environments are available (development, staging, production)?

- Do you have existing test cases or testing tools that you use?

- What’s the timeline and priority for testing the login page?

8. Regulatory and Compliance Issues:

- Are there any legal or regulatory requirements for data protection that need to be considered?

- Does the platform need to comply with standards such as GDPR, HIPAA, or PCI DSS?

By answering these questions, you can better define the scope of your testing and identify specific areas to focus on. This will also help in creating a structured test plan that covers functional, security, performance, and compliance aspects of the login page.

Notice this list does not seem to be complete so you ask again…

Me: This does not seem to be an exhaustive list. Please complete the set of questions

ChatGPT: Absolutely, let’s expand the list to make sure we’re covering a broader range of testing considerations:

9. User Experience and Flow:

- How is the user redirected after a successful login?

- What feedback does the user receive if the login is delayed or unsuccessful?

- Is there a “Forgot Password” feature, and how does it function?

- Does the login page allow for guest checkout or browsing without login?

10. Localization and Internationalization:

- Does the login page support multiple languages?

- Are date, time, and currency formats adjusted according to user locale?

- How does the page handle right-to-left (RTL) languages or other localization needs?

11. Mobile and Cross-Platform Compatibility:

- Is the login page optimized for mobile use?

- Are there any platform-specific issues (iOS vs. Android, different web browsers)?

- How does the login function handle network interruptions or changes (e.g., switching from WiFi to cellular)?

12. Session Management:

- How does the system handle session timeouts?

- Is the user session maintained across different parts of the application after login?

- How are sessions securely terminated after logout or timeout?

13. Test Data and Privacy:

- What data is used for testing the login process?

- How is user data protected during testing?

- Are there specific data handling or privacy concerns that need to be addressed in the test environment?

14. Automated Testing and Continuous Integration:

- Is there a framework in place for automated testing of the login page?

- How are automated tests integrated into the development lifecycle?

- What continuous integration tools are used, and how do they interact with your tests?

15. Non-Functional Requirements and Stress Testing:

- How does the login process perform under stress or load conditions?

- Are there non-functional requirements such as scalability, reliability, or maintainability that need to be tested?

- How does the system recover from a crash or failure during the login process?

16. Regression Testing:

- How often is regression testing performed on the login functionality?

- Are there specific scenarios or previous bugs that need to be regularly retested?

- What is the impact of new features or changes in other parts of the system on the login process?

17. Compliance and Audit Logs:

- Are login attempts and user activities logged for audit purposes?

- How are these logs protected and managed?

- Are there any compliance requirements for monitoring and logging user access?

18. Version Control and Rollback:

- How are changes to the login page managed in version control?

- Is there a rollback plan in case of a problematic update or deployment?

By addressing these comprehensive aspects, you’ll be better equipped to craft detailed test cases that not only cover basic functionality but also ensure robustness, security, usability, and compliance of the login functionality on your e-commerce platform.

This looks like a good list of questions. You may choose to ask further but I stopped here and asked ChatGPT to consolidate both responses into one list…

Me: Now combine ALL the questions from the above responses and please give me a consolidated list of questions

ChatGPT: Certainly! Here’s a consolidated list of questions to guide your testing of the login page on an e-commerce platform:

1. Functionality and Requirements:

- What are the basic functionalities of the login page? (e.g., username/password entry, remember me, password visibility toggle)

- Are there alternative login methods available (social media login, biometrics)?

- What are the mandatory fields on the login page?

- Are there specific requirements or rules for the username and password (length, special characters)?

2. Security Concerns:

- How is the user authentication handled?

- What are the password policies (minimum length, complexity requirements)?

- What happens after multiple failed login attempts? Is there an account lockout or captcha challenge?

- Is there two-factor authentication, and how does it work?

- How are the login credentials stored and transmitted? Is the data encrypted?

3. User Interface and Accessibility:

- What does the login page look like? Can you describe the layout and elements present?

- Is the login page responsive to different devices and screen sizes?

- How accessible is the login page for users with disabilities (screen reader compatibility, keyboard navigability)?

- Are there any specific UI elements like tooltips, error messages, or help links?

4. Performance Expectations:

- What is the expected load time for the login page under normal conditions?

- How many users are expected to use the login function simultaneously?

- Are there performance benchmarks or targets that the login page needs to meet?

5. Integration and Environment:

- How does the login page integrate with other parts of the system (e.g., shopping cart, user profile)?

- What environments does the login page need to support (web, mobile, app)?

- Are there third-party services involved in the login process (e.g., OAuth providers, security services)?

6. Error Handling and Messages:

- What error messages are displayed for invalid inputs or failed logins?

- How user-friendly and informative are the error messages?

- Are error messages logged somewhere for troubleshooting?

7. Testing Logistics:

- What types of testing environments are available (development, staging, production)?

- Do you have existing test cases or testing tools that you use?

- What’s the timeline and priority for testing the login page?

8. Regulatory and Compliance Issues:

- Are there any legal or regulatory requirements for data protection that need to be considered?

- Does the platform need to comply with standards such as GDPR, HIPAA, or PCI DSS?

9. User Experience and Flow:

- How is the user redirected after a successful login?

- What feedback does the user receive if the login is delayed or unsuccessful?

- Is there a “Forgot Password” feature, and how does it function?

- Does the login page allow for guest checkout or browsing without login?

10. Localization and Internationalization:

- Does the login page support multiple languages?

- Are date, time, and currency formats adjusted according to user locale?

- How does the page handle right-to-left (RTL) languages or other localization needs?

11. Mobile and Cross-Platform Compatibility:

- Is the login page optimized for mobile use?

- Are there any platform-specific issues (iOS vs. Android, different web browsers)?

- How does the login function handle network interruptions or changes (e.g., switching from WiFi to cellular)?

12. Session Management:

- How does the system handle session timeouts?

- Is the user session maintained across different parts of the application after login?

- How are sessions securely terminated after logout or timeout?

13. Test Data and Privacy:

- What data is used for testing the login process?

- How is user data protected during testing?

- Are there specific data handling or privacy concerns that need to be addressed in the test environment?

14. Automated Testing and Continuous Integration:

- Is there a framework in place for automated testing of the login page?

- How are automated tests integrated into the development lifecycle?

- What continuous integration tools are used, and how do they interact with your tests?

15. Non-Functional Requirements and Stress Testing:

- How does the login process perform under stress or load conditions?

- Are there non-functional requirements such as scalability, reliability, or maintainability that need to be tested?

- How does the system recover from a crash or failure during the login process?

16. Regression Testing:

- How often is regression testing performed on the login functionality?

- Are there specific scenarios or previous bugs that need to be regularly retested?

- What is the impact of new features or changes in other parts of the system on the login process?

17. Compliance and Audit Logs:

- Are login attempts and user activities logged

for audit purposes?

- How are these logs protected and managed?

- Are there any compliance requirements for monitoring and logging user access?

18. Version Control and Rollback:

- How are changes to the login page managed in version control?

- Is there a rollback plan in case of a problematic update or deployment?

These questions will help ensure a thorough examination of all aspects of the login functionality on your e-commerce platform.

Now that you have the questions, you can ask ChatGPT to generate Test Ideas that address these questions and the risks associated with them…

Me: Now based on the above questions, enlist all the Test ideas that ensure all the questions are answered and all associated risks mitigated.

ChatGPT: Based on the extensive list of questions provided, here’s a comprehensive list of test ideas that can ensure thorough examination and risk mitigation for the login page of an e-commerce platform:

1. Functional Tests

- Standard Login: Test valid username and password combinations to ensure successful login.

- Field Validation: Check for proper validation on all input fields (e.g., format, required fields).

- Alternative Login Methods: Test social media and biometric logins for functionality and security.

- Forgot Password: Verify the forgot password flow including email notifications and security questions.

- Password Visibility Toggle: Ensure the password visibility toggle works correctly without exposing sensitive data.

- Error Handling: Test for appropriate error messages for incorrect inputs or failed login attempts.

2. Security Tests

- Account Lockout: Simulate multiple failed login attempts to test account lockout and captcha functionality.

- Password Policy Enforcement: Check if password complexity and length requirements are enforced.

- Session Security: Test for secure session management ensuring that sessions expire and are securely terminated after logout.

- Data Encryption: Verify that passwords and sensitive data are encrypted when transmitted and stored.

- Two-Factor Authentication: Test the integration and reliability of two-factor authentication mechanisms.

3. UI/UX and Accessibility Tests

- Responsive Design: Ensure the login page is responsive across different devices and screen sizes.

- Accessibility Compliance: Test compatibility with screen readers, keyboard navigation, and adherence to WCAG guidelines.

- UI Elements and Feedback: Verify all UI elements are functional and provide correct feedback for interactions.

4. Performance Tests

- Load Time and Responsiveness: Measure how the login page performs under various network conditions.

- Stress Testing: Simulate a high number of simultaneous users to test system behavior under load.

5. Integration Tests

- Third-Party Services: Test the integration with third-party services for authentication like OAuth providers.

- System Integration: Ensure that login status persists correctly across different sections of the platform after login.

6. Localization and Internationalization Tests

- Multi-language Support: Verify that the login page correctly supports multiple languages and formats.

- Locale-Specific Formats: Test the correctness of localized formats like dates and currency.

7. Mobile and Cross-Platform Compatibility Tests

- Platform-Specific Issues: Test on various platforms (iOS, Android, web browsers) to identify any platform-specific issues.

- Network Interruptions: Evaluate how the login process handles network changes or interruptions.

8. Regression and Continuous Integration Tests

- Automated Regression Tests: Implement automated tests to cover new changes and previous functionalities.

- Continuous Integration: Integrate and monitor automated tests within the CI pipeline to ensure ongoing compatibility and functionality.

9. Compliance and Audit Tests

- Logging and Monitoring: Ensure that all login attempts and user activities are properly logged and comply with regulatory standards.

- Data Protection Compliance: Test to verify compliance with GDPR, HIPAA, PCI DSS, or other relevant standards.

10. Error Recovery and Non-Functional Tests

- Error Message Appropriateness: Check that error messages provide clear, non-technical language suitable for the end-user.

- Recovery from Failures: Simulate different failure scenarios to test the system’s ability to recover without user session or data loss.

By methodically addressing these test ideas, you can ensure comprehensive coverage of the login page’s functionality, security, and performance, helping to mitigate risks and enhance user experience.

I had to make another clarifying request here…

Me: This list of test ideas seems incomplete. Please complete it.

ChatGPT: Let’s enhance the list of test ideas to ensure comprehensive coverage, filling any gaps and adding more detailed scenarios where necessary:

1. Functional Tests

- Input Validation: Verify that all input fields (username, password, captcha) reject invalid formats and respond correctly.

- Session Cookies: Test that cookies are set correctly upon successful login and cleared upon logout.

- Remember Me Function: Ensure the “Remember Me” function sets cookies appropriately and respects user choice across sessions.

- Guest Login/Checkout: Test the functionality and limitations of logging in as a guest, if applicable.

2. Security Tests

- Input Sanitization: Check for SQL injection, XSS, and other security vulnerabilities through input fields.

- HTTPS Enforcement: Verify that the login page is accessible only via HTTPS and not HTTP.

- Security Headers: Ensure security-related HTTP headers are correctly set (e.g., X-Frame-Options, Content-Security-Policy).

- Brute Force Protection: Confirm that protections against brute force attacks (e.g., rate limiting, CAPTCHA after certain attempts) are effective.

3. UI/UX and Accessibility Tests

- Tooltip and Help Text Accuracy: Verify that all tooltips and help texts are correct and helpful.

- Error Message Consistency: Check consistency and appropriateness of error messages across different scenarios (wrong password, field required, session timeout).

4. Performance Tests

- Concurrency Testing: Test how the system handles multiple users logging in at the exact same time.

- Timeouts and Delays: Measure the system’s performance when backend services (like database or authentication server) respond slowly.

5. Integration Tests

- Redirection Post-Login: Test redirection mechanisms after a successful login to ensure users are directed to the intended page or service.

- Service Dependency Handling: Test the login’s dependency on external services by simulating failures of these services to see how well the system handles such situations.

6. Localization and Internationalization Tests

- Right-to-Left Layout Test: Specifically test layouts and UI elements in RTL languages to ensure visual and functional integrity.

- Character Set Handling: Test the acceptance and correct display of user names and passwords in various character sets (e.g., Cyrillic, Chinese).

7. Mobile and Cross-Platform Compatibility Tests

- Touch Interaction: Test usability on touch devices, including button sizes and interactive elements accessibility.

- Network Switching: Test login stability and behavior when switching from mobile data to Wi-Fi and vice versa.

8. Regression and Continuous Integration Tests

- Version Compatibility: Test with previous versions of the application to ensure that updates have not introduced new issues.

- Rollback Scenarios: Test the system’s ability to revert to a previous version in case of a failed update.

9. Compliance and Audit Tests

- Privacy Policy and Terms Link: Ensure that links to privacy policy and terms of service are visible and functional.

- Consent Mechanisms: Test the clarity and functionality of any consent mechanisms for data collection or cookies.

10. Error Recovery and Non-Functional Tests

- Manual Override and Recovery Options: Test available options for recovering from a failed login attempt or system error (e.g., manual admin reset).

- Load Balancer and Failover: Test the effectiveness of load balancers and failover systems in routing traffic and handling failures during peak loads.

11. Additional Test Ideas

- Cross-Site Request Forgery (CSRF) Protection: Test for CSRF vulnerabilities and ensure that the login form is secure against such attacks.

- Session Hijacking: Simulate session hijacking attacks to test the robustness of session security mechanisms.

These expanded test ideas should provide a robust framework for testing the login page, ensuring both functional correctness and security resilience, along with ensuring a positive user experience.

This looks comprehensive however for one last time, I requested ChatGPT to consolidate…

Me: Now combine the above lists today to give a single list with ALL the Test ideas.

ChatGPT: Sure, I’ll combine and organize all the test ideas into a comprehensive list to ensure a thorough testing approach for the login page of an e-commerce platform: