tiers:
-
name: healthcheck
- order: 0
description: >-
First tier to be executed to verify the basic
operations in the VIM.
It creates and configures all tenant network ressources
required by advanced testcases (subnet, network and
router).
+ dependencies:
+ - NO_TENANT_NETWORK: '^(?![tT]rue$)'
run:
name: tenantnetwork1
It creates new user/project before creating and configuring
all tenant network ressources required by a testcase
(subnet, network and router).
+ dependencies:
+ - NO_TENANT_NETWORK: '^(?![tT]rue$)'
run:
name: tenantnetwork2
dependencies:
- DASHBOARD_URL: '^(?!\s*$).+'
run:
- name: tempest_horizon
+ name: tempest_common
args:
- mode: '^tempest_horizon.'
+ mode: '^tempest.scenario.test_dashboard_basic_ops.'
-
name: smoke
- order: 1
description: >-
Set of basic Functional tests to validate the OPNFV scenarios.
testcases:
run:
name: tempest_common
args:
- mode:
- '(?!.*test_incremental_backup)(^cinder_tempest_plugin.)'
+ mode: "(?!.*test_incremental_backup)\
+ (?!.*test_backup_crossproject_admin_negative)\
+ (?!.*test_backup_crossproject_user_negative)\
+ (^cinder_tempest_plugin.)"
option:
- '--concurrency=4'
project_name: functest
criteria: 100
blocking: false
- deny_skipping: true
- tests_count: 222
description: >-
This test case runs a sub group of tests of the OpenStack
Compute testcases.
+ dependencies:
+ - NO_TENANT_NETWORK: '^(?![tT]rue$)'
run:
name: refstack
args:
project_name: functest
criteria: 100
blocking: false
- deny_skipping: true
- tests_count: 278
description: >-
This test case runs a sub group of tests of the OpenStack
Platform testcases.
+ dependencies:
+ - NO_TENANT_NETWORK: '^(?![tT]rue$)'
run:
name: refstack
args:
the OpenStack deployment.
https://github.com/openstack/tempest/blob/18.0.0/tox.ini#L84
run:
- name: tempest_scenario
+ name: tempest_common
args:
mode: '(?!.*\[.*\bslow\b.*\])(^tempest\.scenario)'
option:
run:
name: patrole
args:
- role: 'admin'
- includes:
- - 'image'
- - 'network'
- excludes:
- - 'network.test_flavors_rbac'
- - 'network.test_network_ip_availability_rbac'
- - 'test_networks_multiprovider_rbac'
- - 'test_auto_allocated_topology_rbac'
- - 'test_create_network_provider_segmentation_id'
- - 'test_update_network_provider_physical_network'
- - 'test_update_network_provider_segmentation_id'
- - 'network.test_agents_rbac'
- - 'test_list_metadef_namespaces'
+ roles: 'admin'
+ mode: "(?!.*test_unmanage_snapshot_rbac)\
+ (?!.*test_show_auto_allocated_topology)\
+ (?!.*test_delete_auto_allocated_topology)\
+ (?!.*test_create_network_provider_segmentation_id)\
+ (?!.*compute.test_floating_ips_rbac)\
+ (?!.*test_reset_network)\
+ (?!.*test_create_image_from_volume_backed_server)\
+ (?!.*test_network_ip_availability_rbac.NetworkIpAvailabilityExtRbacTest.test_get_network_ip_availabilities)\
+ (?!.*test_policy_bandwidth_limit_rule_rbac)\
+ (?!.*test_policy_minimum_bandwidth_rule_rbac)\
+ (?!.*test_group_type_specs)\
+ (?!.*test_groups_rbac.GroupTypesV3RbacTest.test_create_group_type)\
+ (?!.*test_groups_rbac.GroupTypesV3RbacTest.test_delete_group_type)\
+ (?!.*test_groups_rbac.GroupTypesV3RbacTest.test_update_group_type)\
+ (?!.*test_quota_classes_rbac)\
+ (?!.*test_volume_metadata_rbac.VolumeMetadataV3RbacTest.test_delete_volume_image_metadata)\
+ (?!.*test_volume_metadata_rbac.VolumeMetadataV3RbacTest.test_list_volumes_details_image_metadata)\
+ (?!.*test_volume_metadata_rbac.VolumeMetadataV3RbacTest.test_show_volume_details_image_metadata)\
+ (?!.*test_volume_metadata_rbac.VolumeMetadataV3RbacTest.test_update_volume_image_metadata)\
+ (?!.*test_volume_types_rbac)\
+ (?=.*[.*\bslow\b.*])\
+ (^patrole_tempest_plugin.tests.api.(compute|image|network|volume))"
+ option:
+ - '--concurrency=4'
-
- case_name: patrole_user
+ case_name: patrole_member
project_name: functest
criteria: 100
blocking: false
run:
name: patrole
args:
- role: 'user'
- includes:
- - 'image'
- - 'network'
- excludes:
- - 'network.test_flavors_rbac'
- - 'network.test_network_ip_availability_rbac'
- - 'test_networks_multiprovider_rbac'
- - 'test_auto_allocated_topology_rbac'
- - 'test_create_network_provider_segmentation_id'
- - 'test_update_network_provider_physical_network'
- - 'test_update_network_provider_segmentation_id'
- - 'network.test_agents_rbac'
- - 'test_list_metadef_namespaces'
+ roles: 'member'
+ mode: "(?!.*test_unmanage_snapshot_rbac)\
+ (?!.*test_show_auto_allocated_topology)\
+ (?!.*test_delete_auto_allocated_topology)\
+ (?!.*test_create_network_provider_segmentation_id)\
+ (?!.*compute.test_floating_ips_rbac)\
+ (?!.*test_reset_network)\
+ (?!.*test_create_image_from_volume_backed_server)\
+ (?!.*test_network_ip_availability_rbac.NetworkIpAvailabilityExtRbacTest.test_get_network_ip_availabilities)\
+ (?!.*test_policy_bandwidth_limit_rule_rbac)\
+ (?!.*test_policy_minimum_bandwidth_rule_rbac)\
+ (?!.*test_group_type_specs)\
+ (?!.*test_groups_rbac.GroupTypesV3RbacTest.test_create_group_type)\
+ (?!.*test_groups_rbac.GroupTypesV3RbacTest.test_delete_group_type)\
+ (?!.*test_groups_rbac.GroupTypesV3RbacTest.test_update_group_type)\
+ (?!.*test_quota_classes_rbac)\
+ (?!.*test_volume_metadata_rbac.VolumeMetadataV3RbacTest.test_delete_volume_image_metadata)\
+ (?!.*test_volume_metadata_rbac.VolumeMetadataV3RbacTest.test_list_volumes_details_image_metadata)\
+ (?!.*test_volume_metadata_rbac.VolumeMetadataV3RbacTest.test_show_volume_details_image_metadata)\
+ (?!.*test_volume_metadata_rbac.VolumeMetadataV3RbacTest.test_update_volume_image_metadata)\
+ (?!.*test_volume_types_rbac)\
+ (?=.*[.*\bslow\b.*])\
+ (^patrole_tempest_plugin.tests.api.(compute|image|network|volume))"
+ option:
+ - '--concurrency=4'
-
case_name: patrole_reader
run:
name: patrole
args:
- role: 'reader'
- includes:
- - 'image'
- - 'network'
- excludes:
- - 'network.test_flavors_rbac'
- - 'network.test_network_ip_availability_rbac'
- - 'test_networks_multiprovider_rbac'
- - 'test_auto_allocated_topology_rbac'
- - 'test_create_network_provider_segmentation_id'
- - 'test_update_network_provider_physical_network'
- - 'test_update_network_provider_segmentation_id'
- - 'network.test_agents_rbac'
- - 'test_list_metadef_namespaces'
+ roles: 'reader'
+ mode: "(?!.*test_unmanage_snapshot_rbac)\
+ (?!.*test_show_auto_allocated_topology)\
+ (?!.*test_delete_auto_allocated_topology)\
+ (?!.*test_create_network_provider_segmentation_id)\
+ (?!.*compute.test_floating_ips_rbac)\
+ (?!.*test_reset_network)\
+ (?!.*test_create_image_from_volume_backed_server)\
+ (?!.*test_network_ip_availability_rbac.NetworkIpAvailabilityExtRbacTest.test_get_network_ip_availabilities)\
+ (?!.*test_policy_bandwidth_limit_rule_rbac)\
+ (?!.*test_policy_minimum_bandwidth_rule_rbac)\
+ (?!.*test_group_type_specs)\
+ (?!.*test_groups_rbac.GroupTypesV3RbacTest.test_create_group_type)\
+ (?!.*test_groups_rbac.GroupTypesV3RbacTest.test_delete_group_type)\
+ (?!.*test_groups_rbac.GroupTypesV3RbacTest.test_update_group_type)\
+ (?!.*test_quota_classes_rbac)\
+ (?!.*test_volume_metadata_rbac.VolumeMetadataV3RbacTest.test_delete_volume_image_metadata)\
+ (?!.*test_volume_metadata_rbac.VolumeMetadataV3RbacTest.test_list_volumes_details_image_metadata)\
+ (?!.*test_volume_metadata_rbac.VolumeMetadataV3RbacTest.test_show_volume_details_image_metadata)\
+ (?!.*test_volume_metadata_rbac.VolumeMetadataV3RbacTest.test_update_volume_image_metadata)\
+ (?!.*test_volume_types_rbac)\
+ (?=.*[.*\bslow\b.*])\
+ (^patrole_tempest_plugin.tests.api.(compute|image|network|volume))"
+ option:
+ - '--concurrency=4'
-
case_name: tempest_barbican
option:
- '--concurrency=2'
+ -
+ case_name: tempest_cyborg
+ project_name: functest
+ criteria: 100
+ blocking: false
+ description: >-
+ It leverages on the tempest plugin containing tests used to
+ verify the functionality of a cyborg installation.
+ run:
+ name: tempest_common
+ args:
+ mode: '^cyborg_tempest_plugin'
+ services:
+ - cyborg
+ option:
+ - '--concurrency=4'
+
-
name: smoke_cntt
- order: 2
description: >-
Set of basic Functional tests to validate the OPNFV scenarios.
testcases:
criteria: 100
blocking: false
deny_skipping: true
- tests_count: 515
+ tests_count: 546
description: >-
This test case runs the Tempest suite proposed by the
Neutron project. The list of test cases is generated by
criteria: 100
blocking: false
deny_skipping: true
- tests_count: 8
+ tests_count: 10
description: >-
This test case runs the Tempest suite proposed by the
Cinder project.
args:
mode: "(?!.*test_incremental_backup)\
(?!.*test_consistencygroups)\
+ (?!.*test_backup_crossproject_admin_negative)\
+ (?!.*test_backup_crossproject_user_negative)\
+ (?!.*test_volume_encrypted.TestEncryptedCinderVolumes)\
+ (?!.*rbac)\
(^cinder_tempest_plugin.)"
option:
- '--concurrency=4'
name: tempest_common
args:
mode: "(?!.*api.identity.v3.test_oauth1_tokens)\
+ (?!.*rbac)\
(?!.*scenario.test_federated_authentication)\
keystone_tempest_plugin."
option:
criteria: 100
blocking: false
deny_skipping: true
- tests_count: 120
+ tests_count: 125
description: >-
This test case runs the Tempest suite proposed by the
Heat project.
name: tempest_heat
args:
mode: "(?!.*functional.test_lbaasv2)\
+ (?!.*functional.test_encryption_vol_type)\
(?!.*RemoteStackTest.test_stack_create_with_cloud_credential)\
(?!.*scenario.test_aodh_alarm)\
(?!.*tests.scenario.test_autoscaling_lb)\
criteria: 100
blocking: false
deny_skipping: true
- tests_count: 1279
+ tests_count: 1269
description: >-
The list of test cases is generated by
Tempest automatically and depends on the parameters of
run:
name: tempest_common
args:
- mode: "(?!.*test_fixed_ips)\
+ mode: "(?!.*admin.test_agents)(?!.*test_fixed_ips)\
(?!.*test_fixed_ips_negative)\
(?!.*test_auto_allocate_network)(?!.*test_floating_ips_bulk)\
(?!.*test_flavors_microversions.FlavorsV255TestJSON)\
(?!.*test_flavors_microversions.FlavorsV261TestJSON)\
(?!.*test_live_migration.LiveAutoBlockMigrationV225Test.test_iscsi_volume)\
+ (?!.*test_live_migration.LiveAutoBlockMigrationV225Test.test_live_block_migration)\
+ (?!.*test_live_migration.LiveAutoBlockMigrationV225Test.test_live_block_migration_paused)\
(?!.*test_live_migration.LiveAutoBlockMigrationV225Test.test_volume_backed_live_migration)\
(?!.*test_live_migration.LiveMigrationTest.test_iscsi_volume)\
+ (?!.*test_live_migration.LiveMigrationTest.test_live_block_migration)\
+ (?!.*test_live_migration.LiveMigrationTest.test_live_block_migration_paused)\
(?!.*test_live_migration.LiveMigrationTest.test_volume_backed_live_migration)\
(?!.*test_live_migration.LiveMigrationRemoteConsolesV26Test)\
(?!.*test_quotas.QuotasAdminTestV257)\
+ (?!.*test_servers.ServersAdminTestJSON.test_reset_network_inject_network_info)\
(?!.*certificates.test_certificates)\
(?!.*test_quotas_negative.QuotasSecurityGroupAdminNegativeTest)\
(?!.*test_novnc)(?!.*test_server_personality)\
(?!.*test_security_groups_negative.SecurityGroupsNegativeTestJSON.test_update_security_group_with_invalid_sg_des)\
(?!.*test_security_groups_negative.SecurityGroupsNegativeTestJSON.test_update_security_group_with_invalid_sg_id)\
(?!.*test_security_groups_negative.SecurityGroupsNegativeTestJSON.test_update_security_group_with_invalid_sg_name)\
+ (?!.*test_create_server.ServersTestFqdnHostnames.test_create_server_with_fqdn_name)\
+ (?!.*test_server_metadata.ServerMetadataTestJSON)\
+ (?!.*test_server_metadata_negative.ServerMetadataNegativeTestJSON.test_delete_metadata_non_existent_server)\
+ (?!.*test_server_metadata_negative.ServerMetadataNegativeTestJSON.test_metadata_items_limit)\
+ (?!.*test_server_metadata_negative.ServerMetadataNegativeTestJSON.test_set_metadata_invalid_key)\
+ (?!.*test_server_metadata_negative.ServerMetadataNegativeTestJSON.test_set_metadata_non_existent_server)\
+ (?!.*test_server_metadata_negative.ServerMetadataNegativeTestJSON.test_set_server_metadata_blank_key)\
+ (?!.*test_server_metadata_negative.ServerMetadataNegativeTestJSON.test_set_server_metadata_missing_metadata)\
+ (?!.*test_server_metadata_negative.ServerMetadataNegativeTestJSON.test_update_metadata_non_existent_server)\
+ (?!.*test_server_metadata_negative.ServerMetadataNegativeTestJSON.test_update_metadata_with_blank_key)\
(?!.*test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_filtered_by_ip_regex)\
(?!.*compute.test_virtual_interfaces)(?!.*compute.test_virtual_interfaces_negative)\
(?!.*compute.test_networks)\
(?!.*test_attach_volume.AttachVolumeMultiAttach)\
(?!.*identity.admin.v2)(?!.*identity.v2)\
+ (?!.*identity.v3.test_access_rules)\
+ (?!.*identity.v3.test_application_credentials.ApplicationCredentialsV3Test.test_create_application_credential_access_rules)\
(?!.*image.v1)\
+ (?!.*image.v2.admin.test_images.ImportCopyImagesTest)\
+ (?!.*image.v2.test_images_negative.ImagesNegativeTest.test_create_image_reserved_property)\
+ (?!.*image.v2.test_images_negative.ImagesNegativeTest.test_update_image_reserved_property)\
+ (?!.*image.v2.test_images_negative.ImportImagesNegativeTest.test_image_web_download_import_with_bad_url)\
+ (?!.*image.v2.test_images.ImportImagesTest)\
+ (?!.*image.v2.test_images.MultiStoresImportImages)\
(?!.*admin.test_dhcp_agent_scheduler)\
(?!.*admin.test_routers_dvr)\
(?!.*test_metering_extensions)(?!.*network.test_tags)\
(?!.*test_routers_negative.DvrRoutersNegativeTest)\
(?!.*test_routers.RoutersIpV6Test.test_create_router_set_gateway_with_fixed_ip)\
(?!.*test_routers.RoutersTest.test_create_router_set_gateway_with_fixed_ip)\
+ (?!.*test_object_services.ObjectTest.test_create_object_with_transfer_encoding)\
+ (?!.*test_encrypted_volumes_extend)\
(?!.*test_group_snapshots.GroupSnapshotsV319Test.test_reset_group_snapshot_status)\
(?!.*test_multi_backend)\
- (?!.*test_snapshot_manage)\
(?!.*test_volume_retype.VolumeRetypeWithMigrationTest)\
(?!.*test_volume_delete_cascade.VolumesDeleteCascade.test_volume_from_snapshot_cascade_delete)\
(?!.*test_volumes_backup.VolumesBackupsTest.test_volume_backup_create_get_detailed_list_restore_delete)\
criteria: 100
blocking: false
deny_skipping: true
- tests_count: 9
+ tests_count: 12
description: >-
The list of test cases is generated by
Tempest automatically and depends on the parameters of
name: tempest_scenario
args:
mode: "\
+ (?!.*test_minbw_allocation_placement)\
+ (?!.*test_unified_limits.ImageQuotaTest.test_image_count_uploading_quota)\
+ (?!.*test_unified_limits.ImageQuotaTest.test_image_stage_quota)\
(?!.*test_volume_boot_pattern.TestVolumeBootPattern.test_boot_server_from_encrypted_volume_luks)\
(?!.*\\[.*\\bslow\\b.*\\])(^tempest.scenario)"
option:
(?!.*test_encrypted_cinder_volumes)\
(?!.*test_minbw_allocation_placement)\
(?!.*test_network_basic_ops.TestNetworkBasicOps.test_router_rescheduling)\
+ (?!.*test_shelve_instance.TestShelveInstance.test_cold_migrate_unshelved_instance)\
(?!.*test_volume_migrate_attached)\
(?!.*test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_cold_migration_revert)\
(?=.*\\[.*\\bslow\\b.*\\])(^tempest.)"
-
name: benchmarking
- order: 3
description: >-
Run several OpenStack performance tools
https://docs.openstack.org/performance-docs/latest/methodologies/tools.html
description: >-
This test case runs a group of Rally jobs used in
OpenStack gating
+ dependencies:
+ - NO_TENANT_NETWORK: '^(?![tT]rue$)'
run:
name: rally_jobs
args:
perform ping connectivity, round trip time measurement
(latency) and TCP/UDP throughput
dependencies:
- - POD_ARCH: '^(?!aarch64$)'
+ - NO_TENANT_NETWORK: '^(?![tT]rue$)'
run:
name: vmtp
is able to deploy OpenStack instances and networks in
different topologies.
dependencies:
- - POD_ARCH: '^(?!aarch64$)'
+ - NO_TENANT_NETWORK: '^(?![tT]rue$)'
run:
name: shaker
-
name: benchmarking_cntt
- order: 4
description: >-
Run several OpenStack performance tools
https://docs.openstack.org/performance-docs/latest/methodologies/tools.html
-
name: vnf
- order: 5
description: >-
Collection of VNF test cases.
testcases:
Clearwater using the Cloudify orchestrator. It also runs
some signaling traffic.
dependencies:
- - DEPLOY_SCENARIO: 'os-.*-nofeature-.*ha'
+ - NO_TENANT_NETWORK: '^(?![tT]rue$)'
run:
name: cloudify_ims
This test case deploys an OpenSource vIMS solution from
Clearwater using the OpenStack Heat orchestrator.
It also runs some signaling traffic.
+ dependencies:
+ - NO_TENANT_NETWORK: '^(?![tT]rue$)'
run:
name: heat_ims
blocking: false
description: >-
This test case is vRouter testing.
+ dependencies:
+ - NO_TENANT_NETWORK: '^(?![tT]rue$)'
run:
name: vyos_vrouter
description: >-
vEPC validation with Juju as VNF manager and ABoT as test
executor.
+ dependencies:
+ - NO_TENANT_NETWORK: '^(?![tT]rue$)'
run:
name: juju_epc
Code Review / functest.git / blobdiff