lkft-android: add cts vts templates for android-mainline projects #247

liuyq · 2023-03-31T17:38:13Z

The cts vts templates added here are generic templates which will be used by many other projects as well

roxell · 2023-04-04T18:33:15Z

maybe do something similar with cts/vts like kselftest and/or ltp does?
look in testcases/(kselftest|ltp)*.yaml and in testcases/master/template-(kselftest|ltp).yaml.jinja2

liuyq · 2023-04-05T15:51:12Z

maybe do something similar with cts/vts like kselftest and/or ltp does? look in testcases/(kselftest|ltp)*.yaml and in testcases/master/template-(kselftest|ltp).yaml.jinja2

I think the current changes are after the reference to the kselftest/ltp files, which parts you think still need to be updated?

roxell · 2023-04-05T17:38:11Z

maybe do something similar with cts/vts like kselftest and/or ltp does? look in testcases/(kselftest|ltp)*.yaml and in testcases/master/template-(kselftest|ltp).yaml.jinja2

I think the current changes are after the reference to the kselftest/ltp files, which parts you think still need to be updated?

Yes, have a master/template-...yaml.jinja2 file and then work from there like kselftest/ltp and others does it.

liuyq · 2023-04-05T18:39:33Z

Yes, have a master/template-...yaml.jinja2 file and then work from there like kselftest/ltp and others does it.

it's OK to move lava_test_plans/testcases/android-cts.yaml and lava_test_plans/testcases/android-cts.yaml to lava_test_plans/testcases/master/template-android-cts.yaml and lava_test_plans/testcases/master/template-android-vts.yaml,
and make the testcases in lkft-android to extends them, if you meant that.

do you have any comments on the content of the files besides the path change?

roxell · 2023-04-12T08:15:45Z

no, thats not what I meant. what I mean is that you can have one file for both vts and cts inside testcases/master/template-android-blah.yaml
and then call that file from testcases/android-vts.yaml and also testcases/android-cts.yaml

roxell · 2023-04-12T08:50:37Z

http://ix.io/4tfv maybe something like this, then it starts to look like kselftest or ltp and others....

roxell · 2023-04-14T06:35:16Z

lava_test_plans/testcases/android-cts.yaml

@@ -0,0 +1,13 @@
+{% extends "testcases/master/template-android-xts.jinja2" %}
+
+{% if TEST_CTS_URL is defined %}


you shouldn't need this if.

roxell · 2023-04-14T06:35:40Z

lava_test_plans/testcases/android-vts.yaml

@@ -0,0 +1,13 @@
+{% extends "testcases/master/template-android-xts.jinja2" %}
+
+{% if TEST_VTS_URL is defined %}


you shouldn't need this if either.

roxell · 2023-04-14T06:36:01Z

lava_test_plans/testcases/master/template-android-xts.jinja2

@@ -0,0 +1,46 @@
+{% extends "testcases/master/template-master.jinja2" %}
+
+{% if xts_test_url is defined %}


not needed here either or

roxell · 2023-04-14T06:36:24Z

lava_test_plans/testcases/master/template-android-xts.jinja2

+{% endblock metadata %}
+
+{% block test_target %}
+{% if TEST_XTS_URL is defined %}


shouldn't be needed.

roxell · 2023-04-14T06:36:45Z

lava_test_plans/testcases/master/template-android-xts.jinja2

+      path: {{xts_test_definition_path}}
+      params:
+        TEST_PARAMS: {% block xts_test_params %}{{xts_test_params}}{% endblock xts_test_params %}
+        TEST_URL: "{{TEST_XTS_URL}}/{{xts_pkg_name}}"


should be test_xts_url and not TEST_XTS_URL

thanks, updated

roxell · 2023-04-14T08:58:54Z

test/test_lava_test_plans.py

@@ -11,7 +11,10 @@

 # all tests all devices


Maybe change this to "# all Linux tests all devices".

roxell · 2023-04-14T09:02:09Z

test/test_lava_test_plans.py

+testcases = [ os.path.basename(d) \
+                for d in glob.glob("lava_test_plans/testcases/*.yaml") \
+                    if not d.startswith("lava_test_plans/testcases/android-")
+            ]


maybe like this

-testcases = [os.path.basename(d) for d in glob.glob("lava_test_plans/testcases/.yaml")]
+testcases = [os.path.basename(d) for d in glob.glob("lava_test_plans/testcases/[!android-].yaml")]

it works with "[!android-]*.yaml"", but I could not read it as [] are normally used for ranges:(

roxell · 2023-04-14T09:03:48Z

lava_test_plans/projects/lkft-android/cts-lkft.yaml

+
+{% set test_timeout = 480 %}
+{% set test_name = "cts-lkft" %}
+{% set cts_test_plan = "cts-lkft" %}


what makes this a cts-lkft test and not just a regular cts test?

non lkft-android project does not need to run this job. or say they might have their plans on what test modules to be run.

roxell · 2023-04-14T09:08:40Z

lava_test_plans/testcases/master/template-android-xts.jinja2

+{## the test_suite_name is a part of the path of the lava result ##}
+{% set xts_test_suite_name = xts_test_suite_name|default(test_name) %}
+{% set xts_test_path = xts_test_path|default("xts-path") %}
+{% set xts_pkg_name = xts_pkg_name|default("android-xts.zip") %}


looks like the path and zip file is more or less the same... maybe drop one variable?

let's keep two variables here, the zip file name is not always like android-cts.zip, there are other places do the renaming for the current setup, maybe some time later the workaround there could be dropped

roxell · 2023-04-14T09:09:07Z

lava_test_plans/testcases/master/template-android-xts.jinja2

+
+{% block metadata %}
+  {{ super() }}
+  {{xts_metadata_prefix}}-url: "{{ xts_test_url }}/android-cts.zip"


xts_pkg_name?

thanks! updated.

roxell · 2023-04-14T09:11:03Z

lava_test_plans/projects/lkft-android/cts-lkft-no-bionic-libcore.yaml

+                --include-filter CtsSystemUiRenderingTestCases
+                --include-filter CtsUsbTests
+                --disable-reboot
+{% endblock xts_test_params %}


I see this as a different android-cts-no-bionic-libcore.yaml test in the testcases directory.
Same with the others.
And not in the projects/lkft-android/ dir.

there will be more workaround files like this to be added later, they are just specific to the lkft-android project or even specific to some android version of some device like the x15 master version here. it's better to put such files under the lkft-android project. otherwise it will make the testcases directory messy with many such one case specific files there.

roxell · 2023-04-21T11:57:06Z

lava_test_plans/testcases/android-cts.yaml

+{% set xts_test_suite_name = xts_test_suite_name|default("cts-lkft") %}
+{% set xts_test_path = xts_test_path|default("android-cts") %}
+{% set xts_pkg_name = xts_pkg_name|default("android-cts.zip") %}
+{% set xts_expect_reboot = xts_expect_reboot|default(false) %}


Not needed. just remove the line

roxell · 2023-04-21T11:58:14Z

lava_test_plans/testcases/android-cts-lkft.yaml

+{% extends "testcases/android-cts.yaml" %}
+
+{% set test_timeout = 480 %}
+{% set test_name = "cts-lkft" %}


what makes this a lkft job?

not get your point, why do you think it is not a lkft job?

what makes these jobs lkft specific or can 'lkft' be dropped from the test_name ?

As you could see there are a list of modules specified here, it's called the cts-lkft test plan, which is specific to the lkft project as it's defined by the lkft project,

So, 1, random cts include-filters enabled that we decided are good to enable, cts-lkft isn't a describing name for this testcase. lkft in the testcase at all isn't describing for what the test does.
lkft is a project not a testcase.

that you have cts-lkft as a test_name in two different test will mess up the results in LAVA right? Two suites with the same name or?

this test case and others are only for the lkft project, and cts-lkft is currently used to described that since the beginning, lkft is a project name, but why it could not be used to define the testcase?

yes, which is required to find the history path in squad automatically.

you are correct.

However, its not needed to have cts-lkft in the job-name, since if you look at how that is built up there is already project which contains lkft-android...

And the filename should probably not be named android-cts-lkft.yaml either.
Yes its the lkft project that have defined this set of include-filters in some way.
Either add a more descriptive name or maybe call it cts-smoke since that's what it looks like.
Since you said that you will add more cts tests, like android-cts-part1, android-cts-part2, android-cts-part3... android-cts-partN.

cts-lkft is the best option for both the job name and the file name here.
smoke is a generic name that should be able to be used for any projects,
This is the cts-lkft plan, for the lkft project, other projects could use it, but it's still the cts-lkft test plan.

roxell · 2023-05-24T11:56:51Z

lava_test_plans/testcases/android-cts.yaml

+{% set xts_test_suite_name = xts_test_suite_name|default("cts-lkft") %}
+{% set xts_test_path = xts_test_path|default("android-cts") %}
+{% set xts_pkg_name = xts_pkg_name|default("android-cts.zip") %}
+{% set xts_metadata_prefix = xts_metadata_prefix|default("cts") %}


please move this file to lava_test_plans/testcases/master/template-android-cts.jinja2
and make it a template.

roxell · 2023-05-24T11:57:09Z

lava_test_plans/testcases/android-vts.yaml

+{% set xts_test_path = xts_test_path|default("android-vts") %}
+{% set xts_pkg_name = xts_pkg_name|default("android-vts.zip") %}
+{% set xts_expect_reboot = xts_expect_reboot|default(true) %}
+{% set xts_metadata_prefix = xts_metadata_prefix|default("vts") %}


please move this file to lava_test_plans/testcases/master/template-android-vts.jinja2
and make it a template.

roxell · 2023-05-24T11:59:02Z

lava_test_plans/testcases/master/template-android-xts.jinja2

+{% set xts_result_format = xts_result_format|default("aggregated") %}
+{% set xts_test_definition_path = xts_test_definition_path|default("automated/android/noninteractive-tradefed/tradefed.yaml") %}
+{% set xts_metadata_prefix = xts_metadata_prefix|default('xts') %}
+{% set xts_version = xts_version|default("xts-version-unknown") %}


A lot of the "dummy defaults" isn't needed here.
Since you said that people should use the template-android-(c|v)ts.jinja2 and not the xtx.jinja2 template.

lava_test_plans/testcases/android-cts-lkft-no-bionic-libcore.yaml

roxell · 2023-05-31T06:44:41Z

lava_test_plans/testcases/android-cts-lkft.yaml

+{% extends "testcases/android-cts.yaml" %}
+
+{% set test_timeout = 480 %}
+{% set test_name = "cts-lkft" %}


So, 1, random cts include-filters enabled that we decided are good to enable, cts-lkft isn't a describing name for this testcase. lkft in the testcase at all isn't describing for what the test does.
lkft is a project not a testcase.

that you have cts-lkft as a test_name in two different test will mess up the results in LAVA right? Two suites with the same name or?

lava_test_plans/testcases/master/template-android-cts.yaml.jinja2

lava_test_plans/testcases/master/template-android-xts.yaml.jinja2

roxell · 2023-06-01T13:52:42Z

lava_test_plans/testcases/android-cts-lkft.yaml

+{% extends "testcases/android-cts.yaml" %}
+
+{% set test_timeout = 480 %}
+{% set test_name = "cts-lkft" %}


you are correct.

However, its not needed to have cts-lkft in the job-name, since if you look at how that is built up there is already project which contains lkft-android...

And the filename should probably not be named android-cts-lkft.yaml either.
Yes its the lkft project that have defined this set of include-filters in some way.
Either add a more descriptive name or maybe call it cts-smoke since that's what it looks like.
Since you said that you will add more cts tests, like android-cts-part1, android-cts-part2, android-cts-part3... android-cts-partN.

roxell · 2023-06-07T06:14:21Z

Please address this comment too.

However, its not needed to have cts-lkft in the job-name, since if you look at how that is built up there is already project which contains lkft-android...

And the filename should probably not be named android-cts-lkft.yaml either.
Yes its the lkft project that have defined this set of include-filters in some way.
Either add a more descriptive name or maybe call it cts-smoke since that's what it looks like.
Since you said that you will add more cts tests, like android-cts-part1, android-cts-part2, android-cts-part3... android-cts-partN.

vishalbhoj · 2023-06-08T12:54:15Z

Please address this comment too.

However, its not needed to have cts-lkft in the job-name, since if you look at how that is built up there is already project which contains lkft-android...

And the filename should probably not be named android-cts-lkft.yaml either. Yes its the lkft project that have defined this set of include-filters in some way. Either add a more descriptive name or maybe call it cts-smoke since that's what it looks like. Since you said that you will add more cts tests, like android-cts-part1, android-cts-part2, android-cts-part3... android-cts-partN.

@liuyq It would be better to break down the android-cts packages as -part1, -part2 etc instead of adding lkft as a test name. There is no relation between the subset vs the name "lkft"

liuyq · 2023-06-08T23:40:51Z

Please address this comment too.
However, its not needed to have cts-lkft in the job-name, since if you look at how that is built up there is already project which contains lkft-android...
And the filename should probably not be named android-cts-lkft.yaml either. Yes its the lkft project that have defined this set of include-filters in some way. Either add a more descriptive name or maybe call it cts-smoke since that's what it looks like. Since you said that you will add more cts tests, like android-cts-part1, android-cts-part2, android-cts-part3... android-cts-partN.

@liuyq It would be better to break down the android-cts packages as -part1, -part2 etc instead of adding lkft as a test name. There is no relation between the subset vs the name "lkft"

@vishalbhoj @roxell basically, there are 3 cts plans will be run for the lkft projects, the full cts, the cts-presubmit, the cts-lkft, the full cts and the cts-presubmit test plans will be run bi-weekly, and the cts-lkft will be run for every build except the bi-weekly builds. the full cts and the cts-presubmit are test plans supported by the cts package by default, while the cts-lkft test plan is a set of modules defined for lkft project, it's not a break down of the full cts plan or the cts-presubmit plan, it should be seen the same thing as the cts or the cts-presubmit, it's a plan name. For the full cts and cts-presubmit, they are broken down as several parts, not lkft there. That's why I think cts-lkft is the best option for both the job names and file names.

roxell · 2023-06-15T13:26:38Z

Please address this comment too.
However, its not needed to have cts-lkft in the job-name, since if you look at how that is built up there is already project which contains lkft-android...
And the filename should probably not be named android-cts-lkft.yaml either. Yes its the lkft project that have defined this set of include-filters in some way. Either add a more descriptive name or maybe call it cts-smoke since that's what it looks like. Since you said that you will add more cts tests, like android-cts-part1, android-cts-part2, android-cts-part3... android-cts-partN.

@liuyq It would be better to break down the android-cts packages as -part1, -part2 etc instead of adding lkft as a test name. There is no relation between the subset vs the name "lkft"

@vishalbhoj @roxell basically, there are 3 cts plans will be run for the lkft projects, the full cts, the cts-presubmit, the cts-lkft, the full cts and the cts-presubmit test plans will be run bi-weekly, and the cts-lkft will be run for every build except the bi-weekly builds. the full cts and the cts-presubmit are test plans supported by the cts package by default, while the cts-lkft test plan is a set of modules defined for lkft project, it's not a break down of the full cts plan or the cts-presubmit plan, it should be seen the same thing as the cts or the cts-presubmit, it's a plan name. For the full cts and cts-presubmit, they are broken down as several parts, not lkft there. That's why I think cts-lkft is the best option for both the job names and file names.

Is there some reason that you don't want to change?
Naming suites after a project, or when things gets merged shouldn't be needed.

liuyq · 2023-06-16T00:09:12Z

Please address this comment too.
However, its not needed to have cts-lkft in the job-name, since if you look at how that is built up there is already project which contains lkft-android...
And the filename should probably not be named android-cts-lkft.yaml either. Yes its the lkft project that have defined this set of include-filters in some way. Either add a more descriptive name or maybe call it cts-smoke since that's what it looks like. Since you said that you will add more cts tests, like android-cts-part1, android-cts-part2, android-cts-part3... android-cts-partN.

@liuyq It would be better to break down the android-cts packages as -part1, -part2 etc instead of adding lkft as a test name. There is no relation between the subset vs the name "lkft"

@vishalbhoj @roxell basically, there are 3 cts plans will be run for the lkft projects, the full cts, the cts-presubmit, the cts-lkft, the full cts and the cts-presubmit test plans will be run bi-weekly, and the cts-lkft will be run for every build except the bi-weekly builds. the full cts and the cts-presubmit are test plans supported by the cts package by default, while the cts-lkft test plan is a set of modules defined for lkft project, it's not a break down of the full cts plan or the cts-presubmit plan, it should be seen the same thing as the cts or the cts-presubmit, it's a plan name. For the full cts and cts-presubmit, they are broken down as several parts, not lkft there. That's why I think cts-lkft is the best option for both the job names and file names.

Is there some reason that you don't want to change? Naming suites after a project, or when things gets merged shouldn't be needed.

No good reason to change it to others.
Again it's a lkft--android specific testcase, not a generic testcase, though it could be used by others.

liuyq · 2023-06-29T09:07:39Z

Hi, @roxell @vishalbhoj
Dropped the addition of android-cts-lkft.yaml, let's see if he remaining things could be merged now.

test/test_lava_test_plans.py

which could be used by default to run the full cts vts package, And these files are generic files, will be reused by other projects via the extentds Signed-off-by: Yongqin Liu <yongqin.liu@linaro.org>

to run the vts-kernel test plan for android platforms Signed-off-by: Yongqin Liu <yongqin.liu@linaro.org>

liuyq force-pushed the android-upstream-cts-vts branch 4 times, most recently from c58b939 to 8f1612b Compare April 3, 2023 05:38

liuyq force-pushed the android-upstream-cts-vts branch from 8f1612b to 0a65ee2 Compare April 14, 2023 06:20

roxell reviewed Apr 14, 2023

View reviewed changes

liuyq force-pushed the android-upstream-cts-vts branch 2 times, most recently from 3a1d16f to a295023 Compare April 14, 2023 08:25

roxell reviewed Apr 14, 2023

View reviewed changes

liuyq force-pushed the android-upstream-cts-vts branch 6 times, most recently from 73059ce to f1017ab Compare April 21, 2023 10:57

roxell reviewed Apr 21, 2023

View reviewed changes

liuyq force-pushed the android-upstream-cts-vts branch from f1017ab to 95bac6e Compare April 21, 2023 14:56

roxell reviewed May 24, 2023

View reviewed changes

liuyq force-pushed the android-upstream-cts-vts branch from 95bac6e to 416f30a Compare May 31, 2023 03:31

roxell reviewed May 31, 2023

View reviewed changes

liuyq force-pushed the android-upstream-cts-vts branch from 416f30a to 56b277c Compare June 1, 2023 08:28

liuyq force-pushed the android-upstream-cts-vts branch from 56b277c to c96f63e Compare June 1, 2023 10:59

roxell reviewed Jun 1, 2023

View reviewed changes

liuyq force-pushed the android-upstream-cts-vts branch from c96f63e to 45ef45e Compare June 2, 2023 02:09

liuyq force-pushed the android-upstream-cts-vts branch 2 times, most recently from ec160ec to bf30d85 Compare June 29, 2023 09:02

roxell reviewed Jul 12, 2023

View reviewed changes

test/test_lava_test_plans.py Show resolved Hide resolved

liuyq added 2 commits July 13, 2023 17:53

testcases: add templates for cts and vts tests

48dde13

which could be used by default to run the full cts vts package, And these files are generic files, will be reused by other projects via the extentds Signed-off-by: Yongqin Liu <yongqin.liu@linaro.org>

testcases: add android vts-kernel templates

2ecc12b

to run the vts-kernel test plan for android platforms Signed-off-by: Yongqin Liu <yongqin.liu@linaro.org>

liuyq force-pushed the android-upstream-cts-vts branch from bf30d85 to 2ecc12b Compare July 13, 2023 09:54

roxell merged commit 5fb7aeb into Linaro:master Jul 13, 2023
2 checks passed

liuyq deleted the android-upstream-cts-vts branch July 13, 2023 16:51

		@@ -0,0 +1,13 @@
		{% extends "testcases/master/template-android-xts.jinja2" %}

		{% if TEST_CTS_URL is defined %}

		@@ -0,0 +1,13 @@
		{% extends "testcases/master/template-android-xts.jinja2" %}

		{% if TEST_VTS_URL is defined %}

		@@ -0,0 +1,46 @@
		{% extends "testcases/master/template-master.jinja2" %}

		{% if xts_test_url is defined %}

lkft-android: add cts vts templates for android-mainline projects #247

lkft-android: add cts vts templates for android-mainline projects #247

Conversation

liuyq commented Mar 31, 2023

roxell commented Apr 4, 2023

liuyq commented Apr 5, 2023

roxell commented Apr 5, 2023

liuyq commented Apr 5, 2023

roxell commented Apr 12, 2023

roxell commented Apr 12, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liuyq May 31, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

roxell commented Jun 7, 2023

vishalbhoj commented Jun 8, 2023

liuyq commented Jun 8, 2023

roxell commented Jun 15, 2023

liuyq commented Jun 16, 2023

liuyq commented Jun 29, 2023

liuyq May 31, 2023 •

edited

Loading