EndToEnd Test Framework
pyOpenPASS
This tool acts as configurable executor for complete sets of configs for the openPASS simulation.
The test framework is located at sim/tests/endToEndTests/pyOpenPASS
.
Prerequisites
Please refer to Installing the Binary Packages for instructions on installing the prerequisites.
Execution
main.py
-s SIMULATION # path to simulation executable, e.g. /openPASS/bin
-m MUTUAL_RESOURCES_PATH # path to mutual config files for all runs, e.g. /examples/common
-r RESOURCES_PATH # path from where configs are retrieved (override common files if necessary)
-c PYOPENPASS_CONFIG.JSON # JSON config for pyOpenPASS
--scope SCOPE_IN_PYOPENPASS_CONFIG # A user defined scope in the config (such as fast, nightly, ...)
-d # Debug some pandas dataframes as csv to /tmp or C:\temp
Configuration
Basic tests: Check executability of configs + Determinism test (1 x n vs n x 1 tests).
Specialized tests: Execute config and check for specific results in the output of the simulator.
Basic Tests
{
"scopes":
{
"THE_SCOPE":
{
"baseConfig":
{
"duration": 60,
"randomSeed": 0,
"invocations": 100
},
"configurations":
[
"CONFIG_UNDER_TEST1",
"CONFIG_UNDER_TEST2"
]
}
}
}
Note
RandomSeed is an optional value. If set, it will override the value given with the configuration.
Parameterized Tests
{
"scopes":
{
"THE_SCOPE":
{
"baseConfig":
{
"duration": 60,
"randomSeed": 0,
"invocations": 100
},
"parameterization":
{
"file": "someConfigFile.xml",
"xpath": "//some/value",
"values": ["value1", "value2", "value3"]
},
"configurations":
[
"CONFIG_UNDER_TEST1",
"CONFIG_UNDER_TEST2"
]
}
}
}
If parameterization specified, the generic tests are executed for each config and each parameter value.
The value is injected into a single configuration file, specified via its filename and an valid xpath.
As values might be contain paths to resources, the variable ${configFolder}
can be used as dynamic placeholder.
Example
With the core being at /OpenPASS/bin
the scope FMU_Test
shall be executed using two different FMUs:
"values": ["${configFolder}/resources/FMUv1.fmu","${configFolder}/resources/FMUv2.fmu" ]
For each parameter, a different config folder configs/FMU_Test_#
is created, where #
is the parameters index.
Before execution, the value of the first parameter ${configFolder}/resources/FMUv1.fmu
is replaced with the absolute path /OpenPASS/bin/configs/FMU_Test_0/resources/FMUv1
.
Specialized Tests
{
"scopes":
{
"THE_SCOPE":
{
"baseConfig":
{
"duration": 30,
"randomSeed": 0,
"invocations": 100
},
"specialized":
{
"description": "A CONCISE DESCRIPTION OF THE SCOPE",
"query": "A VALID QUERY STRING (SEE BELOW)",
"success_rate": 0.9
},
"configurations":
[
"CONFIG_UNDER_TEST1",
"CONFIG_UNDER_TEST2"
]
}
}
}
Note that description
is an optional value, which is currently not used by the framework, but helps to remember, what the test is about.
It is strongly advised to set it, as it might be used in future releases.
Querying Results
Specialized Tests allow to define a query for checking the results of a simulation.
Basic Syntax
[aggregate]([column] | [filter]) [operator] [value]
Aggregate: Everything pandas supports on dataframes, such as pandas.DataFrame.count, min, max, mean
Column: A column on which the aggregate should operate. Columns are given by the simulation outputs cyclic columns, such as
PositionRoute
. AdditionallyAgentId
is made available.Filter: A filter based on pandas.DataFrame.filter syntax using the available columns.
Operator: A comparison operator from the following list: ==, <=, >=, <, >, !=, ~= (approximate). The approximate operator allows
1*e-6 x value
as maximum deviation from value.Value: A number
Example
count(AgentId | PositionRoute >= 800 and Lane != -3) == 0
Using Events in Filter
In order to query for a specific event, use #(EVENT)
within the filter syntax.
Example
count(AgentId | PositionRoute >= 800 and #(Collision) == True) == 0
Event Payload
Each event is associated with a set of triggering entity ids, affected entity ids, and arbitrary key/value pairs (please refer to the openPASS documentation for details). This information is transformed into a “per agent” scope.
In the following the Collision
event is taken as example.
TriggeringEntity
All agents, flagged as triggering become IsTriggering
Query: #(Collision):IsTriggering == True
AffectedEntity
All agents, flagged as affected become IsAffected
Query: #(Collision):IsAffected == True
Key/Value Pairs
If an event publishes additional payload with the key XYZ
, it will can be queried by #(EVENT):XYZ
.
Query: #(Collision):WithAgent
Warning
Keys carrying the event name as prefix, such as in #(Collision):CollisionWithAgent
, will be stripped to Collision:WithAgent
Query Example
count(AgentId | AgentId == 0 and #(Collision):WithAgent == 1) == 0
Using openSCENARIO Events
OpenScenario events are processed in the same manner as regular events (see above).
This allows to query for occurrences of openSCENARIO events with a name specified within the following xpath:
OpenSCENARIO/Story/Act/Sequence/Maneuver/Event/@name
openSCENARIO Event Definition
<Story name="TheStory">
<Act name="TheAct">
<Sequence name="TheSequence" numberOfExecutions="1">
...
<Maneuver name="TheManeuver">
...
<!-- example name "ttc_event"-->
<Event name="ttc_event" priority="overwrite">
...
<StartConditions>
<ConditionGroup>
<Condition name="Conditional">
<ByEntity>
...
<EntityCondition>
<TimeToCollision>
...
</TimeToCollision>
</EntityCondition>
</ByEntity>
</Condition>
</ConditionGroup>
</StartConditions>
</Event>
...
</Maneuver>
</Sequence>
</Act>
</Story>
Example openPASS Output
<Event Time="0" Source="OpenSCENARIO" Name="TheStory/TheAct/TheSequence/TheManeuver/ttc_event">
<TriggeringEntities/>
<AffectedEntities>
<Entity Id="1"/>
</AffectedEntities>
<Parameters/>
</Event>
Query
count(AgentId | #(TheStory/TheAct/TheSequence/TheManeuver/ttc_event) == True ) > 0
Querying Transitions
Sometimes it is necessary to check, whether a transition happened, such as counting agents, passing a certain position.
This can be achieved by shifting individual columns by N
time steps.
Time Shift Syntax
Column-Shift
=> PositionRoute-1
means PositionRoute at one time step earlier
Example Use Case
Counting agents passing PositionRoute == 350
on LaneId == -1
Query
count(AgentId | LaneId == -1 and PositionRoute-1 < 350 and PositionRoute >= 350 ) > 0
Warning
In seldom cases, a result column happens to have a name like Name-N
where N
is an integer.
Querying this column would automatically apply time shifting (default behavior) leading to a parsing error.
In such cases, escape the column name with single quotes (e.g. 'Name-1'
).
Explicit Datatypes
pyOpenPASS uses Pandas DataFrames internally. Pandas will try to detect the datatype of the individual cyclic columns automatically. This won’t fit the user’s intention in some cases, such as when the column holds a semicolon separated list of integers but every list contains just one element. In such cases it is impossible to distinguish between integers and strings based on the data.
For this reason, datatypes can be specified explicitly in the JSON config file:
{
"datatypes": {
"Sensor0_DetectedAgents": "str" // string with "missing value" support
},
"scopes":
{
...
Test Report Generator
Generates an HTML report for results from the pyOpenPASS EndToEndTest Framework tool.
Command
report.py
-r RESOURCE1, RESOURCE2, ... # One or more resource folders
-c PYOPENPASS_CONFIG.JSON # JSON config for pyOpenPASS
--scope SCOPE1, SCOPE2, ... # One or more scopes to be considered
--results SIMULATOR_RESULT_PATH # Path with results from simulator (should be the same for all scopes)
--output PATH_FOR_REPORT # Path where the report should be written to