A - 09 Verkeer, Vervoer en Luchtkwaliteit (40) bijl 4. WSP Amsterdam Metro C
Tekst van het document
Amsterdam Metro
CBTC COMMISSIONING REVIEW Report
REV 1.0 21ST DEC 2021
CONFIDENTIAL
Amsterdam Metro
CBTC COMMISSIONING REVIEW Report
TYPE OF DOCUMENT (VERSION) CONFIDENTIAL
PROJECT NO. 70089825 OUR REF. NO. REV 1.0 DATE: 21ST DEC 2021
WSP 4th Floor 6 Devonshire Square London EC2M 4YE Phone: +44 20 7337 1700 Fax: +44 20 7337 1701 WSP.com
CONFIDENTIAL
QUALITY CONTROL Issue/revision
First issue
Remarks
Formal Release
Date
21/12/21
Prepared by
Sanjeev Rathi
Revision 1
Revision 2
Revision 3
Signature Checked by
Stuart Dickson Digitally signed by Dickson, Stuart (UKSMD003) DN: cn=Dickson, Stuart (UKSMD003), ou=Active, email=Stuart.Dickson1@wsp.com Date: 2022.01.07 13:09:16
Signature Authorised by
Pradeep Vasudev
Signature
Pradeep Vasudev
Project number
70089825
Report number
Rev1.0
Digitally signed by Pradeep Vasudev DN: cn=Pradeep Vasudev, c=GB, o=WSP UK Ltd, ou=Director International Projects, email=pradeep.vasudev@wsp.com Date: 2022.01.07 13:31:58
File reference
CBTC COMMISSIONING REVIEW Project No.: 70089825 | Our Ref No.: Rev 1.0 Amsterdam Metro
CONFIDENTIAL | WSP 21st Dec 2021
CONTENTS
1
BACKGROUND
1
2
PURPOSE OF THIS REPORT
2
3
METHODOLOGY FOR THE PRODUCTION OF REPOT
3
4
REVIEW OF THE PROJECT
4
4.1
REVIEW OF GLOBAL HOLDS
4
4.2
REGRESSION TESTING
5
4.3
TEST COVERAGE
5
4.4
VERIFICATION TESTING
6
4.5
TEST PLANNING AND COMMISSIONING STRATEGY
7
4.6
SOFTWARE RELEASE STRATEGY AND ASSURANCE
8
4.7
OPERATIONAL RESTRICTONS AND DEFECTS RETIREMENT PLAN
9
4.8
RELIABILITY DATA
10
4.9
LESSONS LEARNED
10
5
RESPONSE TO THE QUESTIONS
11
6
CONCLUSION
12
7
APPENDICES
13
7.1
APPENDIX I: SUMMARY OF GLOBAL HOLDS
13
7.2
APPENDIX II: NEW OPERATIONAL RESTRICTIONS
14
7.3
APPENDIX III: PROJECT REVIEW FOR RESTART OF COMMISSIONING
15
CBTC COMMISSIONING REVIEW Project No.: 70089825 | Our Ref No.: Rev 1.0 Amsterdam Metro
CONFIDENTIAL | WSP 21st Dec 2021
1
BACKGROUND The City of Amsterdam has been working on the implementation of a new signalling and control system since 2012. The first part of the project has been deployed in 2018 on the new North South Line. In February’21 it started the deployment of CBTC on the rest of the network. This network is a fully integrated network with bundled operations, which means there are no separate tracks for separate lines. Therefore, it was not possibility to introduce the system line by line. To avoid a big bang Amsterdam established a ‘staged commissioning’ strategy. As part of the commissioning strategy, during a set period the new system and the legacy system are alternately used, i.e. switchable. On the 28th of February 2021 Amsterdam Metro started to use the system on weekends and gradually extending the number of days per week. Apart from encountering some reliability issues during this period, there were eight incidences of Global Hold1 between April’21- Sept’212, which led to reverting of control back to legacy signalling system on 25th Sept’21. Only the North South line continued to operate on the CBTC system.
Following concerns resulting from multiple incidences of a Global Hold, Amsterdam metro has asked WSP to carry out the project review to answer following questions; Q1 - Has Amsterdam taken, based on your experience, reasonable steps to restart commissioning? Q2 - Do you see any reason to believe a successful implementation of the system on the Amsterdam network is not possible? If so, are there alternatives to the CBTC system available that have more potential success? Q3 - What’s your opinion about the current implementation strategy and in particular the Staged Commissioning approach? Q4 - Are there any recommendations that you propose for adjusting or improving the current implementation strategy? WSP UK submitted its report to Amsterdam Metro project team on 21/10/2021 to answer the Question 1, which was considered as priority to restart CBTC commissioning on rest of the network whilst the N-S line continued to operate on CBTC. The CBTC was recommissioned on 24th Oct’21. The report in response to question 1 is attached at Appendix III. The other questions are responded in this report.
(NB: 1Global Hold is a controlled shutdown process to gracefully bring the railway in a safe state. Total 11 Global Holds between April’21 – Nov’21 – see Appendix I).
2
CBTC COMMISSIONING REVIEW Project No.: 70089825 | Our Ref No.: Rev 1.0 Amsterdam Metro
CONFIDENTIAL | WSP 21st Dec 2021 Page 1 of 15
2
PURPOSE OF THIS REPORT The purpose of this report is to provide response to following outstanding questions raised by Amsterdam metro. ▪
Do you see any reason to believe a successful implementation of the system on the Amsterdam network is not possible? If so, are there alternatives to the CBTC system available that have more potential success?
▪
What’s your opinion about the current implementation strategy and in particular the Staged Commissioning approach?
▪
Are there any recommendations that you propose for adjusting or improving the current implementation strategy?
CBTC COMMISSIONING REVIEW Project No.: 70089825 | Our Ref No.: Rev 1.0 Amsterdam Metro
CONFIDENTIAL | WSP 21st Dec 2021 Page 2 of 15
3
METHODOLOGY FOR THE PRODUCTION OF REPOT For project review, WSP has taken a risk-based approach based on its experience on similar complex signalling programmes. This involved looking various aspects, including; (a)
Review of Global Holds
(b)
Regression testing
(c)
Test coverage and verification testing
(d)
Test planning and commissioning strategy
(e)
Software release strategy and assurance
(f)
Operational restrictions and defect retirement plan
(g)
Reliability data
(h)
Lessons learned
For this purpose, WSP carried out number of interviews with targeted questions involving people from project, operations and supplier team. ▪
•
Following people were involved in review from WSP o
Pradeep Vasudev - WSP’s Global Systems Integration Expert
o
Stuart Dickson - Major Projects Rail Systems and Systems Integration Expert
o
Sanjeev Rathi - CBTC, ETCS and Rail Systems Expert
Following people from the project were interviewed by WSP team Programme Delivery o
Gerard Scheffrahn
o
Pelle de Wit
Project T&C Manager o
Carel Ilgen
Project IT & Technology Manager o
Gert van der Beek
Operations o
Wim van Oort
o
Frans Cramer
Project Operational Expert o
Greet Burkels
ALSTOM o
Paul VISSER
o
Laurent Duret
CBTC COMMISSIONING REVIEW Project No.: 70089825 | Our Ref No.: Rev 1.0 Amsterdam Metro
CONFIDENTIAL | WSP 21st Dec 2021 Page 3 of 15
4
REVIEW OF THE PROJECT WSP review of the project is based on the information supplied during the interviews and subsequently as part of written questions.
4.1
REVIEW OF GLOBAL HOLDS The frequency of Global Holds has been a cause of concern for Amsterdam metro. There have been 11 incidences of Global Holds between Feb’21 (commissioning of CBTC) and Nov’21 [Appendix I] Number of root causes/ trigger points have been attributed by ALSTOM to these Global Holds, which include; •
CPU Overload
•
Memory Overload
•
Automatic settings overwritten by conflicting manual intervention
•
Data base corruption
•
Train tracking module moving block
•
Train loss of primary localisation
•
Coupled operation of 2 or more vehicles across the zone controller boundary
The number of trigger points causing Global Holds do raise a question about the vulnerability of the system and there is a need for an in-depth analysis of the system architecture and design principles. A few examples of vulnerability of the system include; -
database corruption because of manual intervention by traffic operator during ATS automations (ref. Global Hold on 20/11/21)
-
CPU and memory overload because of train tracking functionality in moving block versus coupled operation of 2 or more trains
A good system design is expected to deal with operator error in a graceful manner rather causing it to crash. There is also a question about the adequacy of test procedures to identify some of the issues during testing, as an example, an issue caused by manual intervention during ATS automation could be identified by negative testing 3. A good system design should also include notification in the form of audible -visual warnings about health of the system to allow timely corrective action. Recommendations 1. In-depth review of system design and architecture to confirm robustness of the system against inadvertent actions and operational errors. 2. Perform negative testing to test the robustness of the system during acceptance test. 3. Audible -visual warning about health of the system (e.g. CPU memory usage) to allow timely corrective action. 3
NB: It is recognised that some of the issues are difficult to simulate on the rigs especially the timing issues. Assessment of capability of test rigs would help in identifying test environment limitations and in defining alternate means of testing.
CBTC COMMISSIONING REVIEW Project No.: 70089825 | Our Ref No.: Rev 1.0 Amsterdam Metro
CONFIDENTIAL | WSP 21st Dec 2021 Page 4 of 15
4.2
REGRESSION TESTING There is evidence of regression as can be seen from the emergence of new issues and recurrence of old issues, for example; -
ARS not working
-
Deletion of trips in the production plan
-
Immediate departure
ALSTOM’s current approach of testing is based on a list of test cases under 3 categories; •
Product Changes (standard test cases)
•
Patches for temporary mitigations (test of the fixes, no test of regression)
•
New baseline deliveries (FAT procedures, FIPV tests and T&C based on Impact analysis)
Product changes and new baseline deliveries seem to follow standard test cases and FAT procedures, whereas the testing for temporary patches is based on subset of test cases. There seems to be focus on testing the fixes and Alstom does not explain how the adverse effects of a change are tested. It is not uncommon to see the regression due to change in one part of the software manifesting in other interfacing parts of the sub-system, especially if different software teams are involved in the software update. Although ALSTOM do not carry out Critical Code Review (CCR) by default, but it seems following a Global Hold in September’21, there is increased focus on CCR. As the reliability of the system continues to be a cause of concern in WSP’s view Critical Code Review is a good tool to control emergence of new issues. Also, in our view ALSTOM should also review their test cases to adapt them to Amsterdam metro specific requirements in terms of operational scenarios and vulnerability of the system experienced during Global Holds. Recommendations: 4. ALSTOM should review their regression test strategy. Critical Code Review (CCR) should be considered as standard across all future software releases 5. ALSTOM to review their test cases to adapt them to Amsterdam metro specific requirements in terms of operational scenarios and vulnerability of the system experienced during Global Holds.
4.3
TEST COVERAGE There is a heavy reliance on the off-site testing on this project, which is a good way to reduce the site access and operational disruptions. However, it does require an assurance on adequacy of the test coverage both in terms of capability of rigs and adequacy of test specifications. It seems ALSTOM have carried out the analysis of rigs capability as part of their V&V process and have identified following limitations; •
Simulation of Zone Controller data linked to coupled trains
CBTC COMMISSIONING REVIEW Project No.: 70089825 | Our Ref No.: Rev 1.0 Amsterdam Metro
CONFIDENTIAL | WSP 21st Dec 2021 Page 5 of 15
•
Slip & slide characteristics on the rigs are not true representative of the RS braking system
The current test regime is based on normal day operation scenarios, which means some of the issues that have been recently encountered are difficult to find as part of normal testing. Although not everything can be uncovered in off-site testing. However, using experience of Global Holds over last 10 months, it will be useful to carry out testing by stressing the system on the rigs and using unusual scenarios in order to test the resilience of the system. Recommendations: 6. ALSTOM should review the test cases and test strategy to cover abnormal operation scenarios to test the resilience of the system and bug finding. We believe that operators’ input into test cases would help. 7. Considerations should be given for negative testing to assess the system behaviour against the inadvertent / incorrect operator actions e.g. conflicting commands by the operator.
4.4
VERIFICATION TESTING For verification testing ALSTOM use the number of steps under the test case as success criteria rather than pass/fail against a test case itself. This does not provide a true reflection test success. Each test case has a specific objective, in our view the pass/fail criteria should be against a test case and not based on number of steps. If the test fails a particular step during execution of a test case, an explanation should be given to close it or a defect should be raised. Recommendation: 8. Pass/ fail criteria should be based against test cases
CBTC COMMISSIONING REVIEW Project No.: 70089825 | Our Ref No.: Rev 1.0 Amsterdam Metro
CONFIDENTIAL | WSP 21st Dec 2021 Page 6 of 15
4.5
TEST PLANNING AND COMMISSIONING STRATEGY Amsterdam metro project test planning and commissioning strategy for new software release is based on; • ALSTOM Testing (i)
ALSTOM off-site testing at various rigs (NB: For important baseline updates FAT is carried out on the simulator in Paris. Key users i.e. traffic operators are involved).
(ii)
Onsite testing - usually from Friday to Saturday night (4-hour possession) and Saturday to Sunday night (4-hour possession)
• Integration Testing by the project (iii)
6- 8 night of testing with test environment server (over & back testing using “great switch”)
• Go- live and switchover (iv)
Go live with test environment server – 4 days of running (normal operation)
(v)
Switch over to production server (normal operation)
The overall test and commissioning strategy follow a sound principle of controlled change process to the live railway. However, given the experience so far, it is worth looking for improvements in execution of above tests. Recommendations: 9. Involving Operations staff to witness off-site testing 10. Allow a window (4 to 8 hrs) for operations staff “to play with the system” as part of FAT following ALSTOM formal off-site testing 11. Consider enhancing the integration tests to include stress testing of the system
CBTC COMMISSIONING REVIEW Project No.: 70089825 | Our Ref No.: Rev 1.0 Amsterdam Metro
CONFIDENTIAL | WSP 21st Dec 2021 Page 7 of 15
4.6
SOFTWARE RELEASE STRATEGY AND ASSURANCE As per below, project has 8-software releases planned in a space of 12 months. This means a software release every 6 to 8 weeks, which in our view seems bit too frequent. Apart from being a drain on scarce resources, it also risks the quality of the software, for example; -
Risk of parallel software development tracks
-
Reducing the capacity of software build
-
Squeezing the process time could undermine quality and increase risk of regression
SW version
Release date/ planned Remarks release
BL4.8.1.13-11
24/04/2021
Includes solution for wrong predictions in the passenger information
BL4.8.1.13-13
01/05/2021
Includes solution for split label issues
BL4.8.1.13-??
05/06/2021
Includes solution for wrong predictions in the passenger information
BL5.2.1.6
08/08/2021
Includes solution for trip still running from the previous day
BL5.2.1.6-15
24/10/2021
Includes solution for ATS CPU peaks and redundancy failure
BL5.2.17
18/12/2021
Includes solutions for global holds 29/10, 18/11 (infinite loops) & 20/10 (incorrect manual routing)
BL 5.2.1.8
Feb ’22
Includes solutions for LOP’s, mainly related to label issues, ARS and EB’s
BL5.2.1.9
April ‘22
Includes solutions for LOP’s, mainly related to label issues, ARS and EB’s
Current strategy of software updates in quick succession isn’t helping the rate of defects burn down, which remains considerably low. An improvement in software production quality would help in accelerating the burndown rate. It is understood that some of the software releases could be driven by emergency fixes. Recommendation: 12. Project should review of software release strategy considering associated risks
CBTC COMMISSIONING REVIEW Project No.: 70089825 | Our Ref No.: Rev 1.0 Amsterdam Metro
CONFIDENTIAL | WSP 21st Dec 2021 Page 8 of 15
4.7
OPERATIONAL RESTRICTONS AND DEFECTS RETIREMENT PLAN There is a concern about the slow defects burn down rate. As on 01/12/2021, the breakdown of LOPs (presented by Ops team); Major issues B = 140 Minor issues C = 115 Cosmetic issues D = 15 The number of major issues seem on higher side. It is our understanding whilst old defects are solved but are being replaced by new ones, although it is not unusual to find new issues during early years of operation for a complex railway like Amsterdam metro but it does require a closer attention. NB: This is also supported by the data presented by ALSTOM for the period 01/03/2021 – 01/12/2021, which includes 108 LOPs, of which 25 are newly added and only 20 are either closed or rejected. The remaining are either on hold (1) or in progress (52). The average time to retire defect from opening and closing the LOP is 95 days. 6 new ORs added since last software release [Appendix II] Defect retirement over the life of the project to date (6 Dec’21)
We recommend project to review its strategy to accelerate the defect retirement. Recommendations: 13. Consider constituting a task force (including reps from Ops, project, and ALSTOM) to review prioritisation 14. ALSTOM should enhance their effort in investigating and developing the solution to fix the issues. 15. Review software release strategy as having too frequent software release might slowdown the pace of defect retirement as it drains the resources for assuring each software release. 16. Managing the risk of regression to contain emergence of new defects
CBTC COMMISSIONING REVIEW Project No.: 70089825 | Our Ref No.: Rev 1.0 Amsterdam Metro
CONFIDENTIAL | WSP 21st Dec 2021 Page 9 of 15
4.8
RELIABILITY DATA This an area that needs to be closely looked at. From discussion with operations team, it appears there is a variance between reliability data produced by ALSTOM and the performance data collated by the Operations team. As an example, a failure considered as SAF (service affecting failure) by operations team may not be counted for SAF in ALSTOM periodic RAMS analysis. The quality of the reliability data is extremely important to drive the real improvement as well as managing the expectations of the stakeholders. In case of above example, it could be argued that any failure that has a potential to cause SAF, irrespective of whether SAF occurred or not should be counted as SAF for the purpose of severity and defect retirement. Recommendations: 17. Having one source of reliability data as a single source of truth 18. Sharing the raw data between the operations and ALSTOM team to drive the common understanding on reliability figures 19. Having heat map that includes frequency of occurrence and severity to prioritise the defects retirement 20. Identification of location and Asset specific faults to help the investigation in the right direction
4.9
LESSONS LEARNED There are positive indications that the Project has actively engaged in lessons learned recently and actively working to implement the improvements to alleviate the situation on the ground both in terms of containing the failures and managing the downtime during the failures. For example; •
ALSTOM on-site support in the TCC is now available 20 hrs a day to enable faster recovery time in case of failures (NB: Notification and remote access to re-boot the server to ALSTOM maintenance team is being planned and should be available BL 5.2.1.7 to be deployed in Dec’21)4
•
•
ALSTOM have also taken on-board following process improvements; o
Quick response team for having the faster response time in case of Global Holds and obtaining better & accurate information for investigation
o
Implementation of a complete day of testing based on actual timetable (normal day in Amsterdam)
o
Robustification of product tests by adding more test cases based on the problems found, design review and V&V strategy
There is a good example of collaboration between various parts of the project team to address the issues related to emergency braking
(NB: 4 Implemented on 18th Dec’21)
CBTC COMMISSIONING REVIEW Project No.: 70089825 | Our Ref No.: Rev 1.0 Amsterdam Metro
CONFIDENTIAL | WSP 21st Dec 2021 Page 10 of 15
5
RESPONSE TO THE QUESTIONS Based on the findings, the response to following questions raised by Amsterdam metro is as under; Questions: ▪
Do you see any reason to believe a successful implementation of the system on the Amsterdam network is not possible? If so, are there alternatives to the CBTC system available that have more potential success?
▪
What’s your opinion about the current implementation strategy and in particular the Staged Commissioning approach?
▪
Are there any recommendations that you propose for adjusting or improving the current implementation strategy?
We have no reason to believe that CBTC is not the right solution in case of Amsterdam metro. CBTC is a standard solution for metro environment. The CBTC solution has been applied to London underground 4 LM project, which has a very similar network to Amsterdam, where we have multiple lines following same track (guideway). The solution has been applied in a brownfield environment. The main difference between Amsterdam metro and London underground is the system architecture and migration strategy. On London underground the network is divided into number of Migration Areas and is supported by system architecture as part of design. This takes longer and is more expensive but there are benefits as failure can be contained to smaller part of the network. It also allows reliability growth in a controlled manner. In case of Amsterdam metro, whilst there is a benefit of faster deployment but failures in some cases could impact large part of the network, Amsterdam metro’s risk profile is also affected by special operational requirements such as coupled trains and especially with the required flexibility to couple and uncouple during operational window. This is further complicated by mix use of old and new rolling stock. In case on London underground, these variables are absent.
As we understand, Amsterdam metro are in the process of deployment of new rolling stock, once the new rolling stock is deployed on whole network, we believe Amsterdam metro should see reliability improvement. We believe that as a long-term measure, it would be beneficial for Amsterdam metro to review the system architecture to make it more resilient. For example, re-defining the zone controller boundaries and making improvements in system design at ATS level. This would require proper option engineering to determine the cost- effective solution. As mentioned in our letter dated 18 th November’21, encountering new issues during implementation of an integrated control system like CBTC on a complex network in a brownfield environment like Amsterdam is not an unexpected phenomenon. It can take several years once the system is fully operational to identify all the technical issues and it is common practice to manage these as a combination of operational procedures, maintenance regime, training, and technical solution. This requires continuous proactive monitoring and management between the project/supplier and operator to quickly identify issues that impact operations and implement short-term and longer-term solutions.
CBTC COMMISSIONING REVIEW Project No.: 70089825 | Our Ref No.: Rev 1.0 Amsterdam Metro
CONFIDENTIAL | WSP 21st Dec 2021 Page 11 of 15
6
CONCLUSION We believe that CBTC is the right solution for Amsterdam metro and there should be improvement in reliability once new rolling stock is deployed. Further improvements can be made by enhancing the quality of the software and improving the system resilience to the abnormal operational conditions. It is not surprising to see new issues during early phase of CBTC implementation in a brownfield environment. As we have stated earlier that while most of the technical issues are resolved during systems testing but some issues arising from operational conditions and change in environment are likely to escape normal system testing and would manifest during actual operation. We believe that recommendations in section 4 of this report along with the recommendations in our report dated 21/10/2021 (Appendix III) would help in driving these improvements.
CBTC COMMISSIONING REVIEW Project No.: 70089825 | Our Ref No.: Rev 1.0 Amsterdam Metro
CONFIDENTIAL | WSP 21st Dec 2021 Page 12 of 15
7
APPENDICES
7.1
APPENDIX I: SUMMARY OF GLOBAL HOLDS Date
Global Hold Description
Fix Plan
Global Holds Feb-June'21 12/04/2021
Issue at regulation /prediction algorithm module level
BL5.2.1.6
25/04/2021
BL5.2.1.6
31/05/2021
Issue at regulation /prediction algorithm module level Working Instruction to avoid tripcode 2-issues neglected (Traffic Controller) Due to a trip remaining running from a TooAP that occurred on 30/05
14/06/2021
Trip still running from the previous day
BL5.2.1.6
16/05/2021
N/A BL5.2.1.6
Global Holds in Sept'21 GTW-DRIS (Siemens PIS ) causing ATS CPU peaks and 13/09/2021 freezes
BL5.2.1.6-15
24/09/2021
BL5.2.1.6-15
25/09/2021
ATS CPU peaks and redundancy failure TooAP Zone Controller 3 goes in safe state – Staff focussed erroneously on ATS
Global Holds since CBTC recommissioning in Oct'21 29/10/2021 Infinite loop entered at module level after ghost train / false occupancy 18/11/2021 Tracking issue - transition from Fixed Block to Moving Block where train was stopped on the ‘Static Dark Zone’ and switched off. Triggering the infinite loop 20/11/2021 Incorrect manual routing of the train causing ATS system to crash
CBTC COMMISSIONING REVIEW Project No.: 70089825 | Our Ref No.: Rev 1.0 Amsterdam Metro
N/A
BL5.2.1.7
BL5.2.1.7 BL5.2.1.7
CONFIDENTIAL | WSP 21st Dec 2021 Page 13 of 15
7.2
APPENDIX II: NEW OPERATIONAL RESTRICTIONS Six operational restrictions added since last sw release (BL 5.2.1.6):
CBTC COMMISSIONING REVIEW Project No.: 70089825 | Our Ref No.: Rev 1.0 Amsterdam Metro
CONFIDENTIAL | WSP 21st Dec 2021 Page 14 of 15
7.3
APPENDIX III: PROJECT REVIEW FOR RESTART OF COMMISSIONING
Amsterdam Metro_Project Review for restart of commissioning _Rev 1.pdf
CBTC COMMISSIONING REVIEW Project No.: 70089825 | Our Ref No.: Rev 1.0 Amsterdam Metro
CONFIDENTIAL | WSP 21st Dec 2021 Page 15 of 15
4th Floor 6 Devonshire Square London EC2M 4YE wsp.com
CONFIDENTIAL