Hello,
Currently trying to repair an OT-2 where the robot server won’t start seemingly due to the the fact that the pi can’t communicate with the Smoothieboard. I though this was a hardware issue with the board so I just replaced it with a new one, though I’m still having issues. Below is a log of process for starting robot server, hopefully it can be made sense of:
Feb 27 20:18:35 0402e3 systemd[1]: Starting Opentrons Robot HTTP Server…
Feb 27 20:19:29 0402e3 uvicorn[1161]: INFO: Started server process [1161]
Feb 27 20:19:29 0402e3 uvicorn[1161]: INFO: Waiting for application startup.
Feb 27 20:19:29 0402e3 uvicorn[1161]: /data/robot_settings.json not found. Loading defaults
Feb 27 20:19:29 0402e3 uvicorn[1161]: Application startup complete.
Feb 27 20:19:29 0402e3 uvicorn[1161]: Uvicorn running on unix socket /run/aiohttp.sock (Press CTRL+C to quit)
Feb 27 20:19:29 0402e3 uvicorn[1161]: Stopped opentrons-gpio-setup.
Feb 27 20:19:29 0402e3 uvicorn[1161]: /data/robot_settings.json not found. Loading defaults
Feb 27 20:19:29 0402e3 opentrons-api[1161]: API server version: 7.0.1
Feb 27 20:19:29 0402e3 opentrons-api[1161]: Robot Name: opentrons
Feb 27 20:19:30 0402e3 opentrons-api[1161]: Connecting to motor controller at port /dev/ttyAMA0
Feb 27 20:19:30 0402e3 uvicorn[1161]: Using directory /data/opentrons_robot_server for persistence.
Feb 27 20:19:30 0402e3 opentrons-api[1161]: /data/robot_settings.json not found. Loading defaults
Feb 27 20:19:30 0402e3 opentrons-api[1161]: Registering Central Routing Board Revision GPIOs
Feb 27 20:19:30 0402e3 opentrons-api[1161]: Configuring GPIOs by central routing roard revision
Feb 27 20:19:30 0402e3 opentrons-api[1161]: Setting up GPIOs
Feb 27 20:19:30 0402e3 uvicorn[1161]: BEGIN (implicit)
Feb 27 20:19:30 0402e3 uvicorn[1161]: BEGIN
Feb 27 20:19:30 0402e3 uvicorn[1161]: [raw sql] ()
Feb 27 20:19:30 0402e3 uvicorn[1161]: PRAGMA main.table_info(“migration”)
Feb 27 20:19:30 0402e3 uvicorn[1161]: [raw sql] ()
Feb 27 20:19:30 0402e3 uvicorn[1161]: PRAGMA main.table_info(“protocol”)
Feb 27 20:19:30 0402e3 uvicorn[1161]: [raw sql] ()
Feb 27 20:19:30 0402e3 uvicorn[1161]: PRAGMA main.table_info(“analysis”)
Feb 27 20:19:30 0402e3 uvicorn[1161]: [raw sql] ()
Feb 27 20:19:30 0402e3 uvicorn[1161]: PRAGMA main.table_info(“run”)
Feb 27 20:19:30 0402e3 uvicorn[1161]: [raw sql] ()
Feb 27 20:19:30 0402e3 uvicorn[1161]: PRAGMA main.table_info(“action”)
Feb 27 20:19:30 0402e3 uvicorn[1161]: [raw sql] ()
Feb 27 20:19:30 0402e3 uvicorn[1161]: COMMIT
Feb 27 20:19:30 0402e3 uvicorn[1161]: BEGIN (implicit)
Feb 27 20:19:30 0402e3 uvicorn[1161]: BEGIN
Feb 27 20:19:30 0402e3 uvicorn[1161]: [raw sql] ()
Feb 27 20:19:30 0402e3 uvicorn[1161]: PRAGMA main.table_info(“run”)
Feb 27 20:19:30 0402e3 uvicorn[1161]: [raw sql] ()
Feb 27 20:19:30 0402e3 uvicorn[1161]: SELECT migration.id, migration.created_at, migration.version
FROM migration ORDER BY migration.version DESC
Feb 27 20:19:30 0402e3 uvicorn[1161]: [generated in 0.00212s] ()
Feb 27 20:19:30 0402e3 uvicorn[1161]: Database has schema version 2. no migrations needed.
Feb 27 20:19:30 0402e3 uvicorn[1161]: SELECT analysis.id, analysis.completed_analysis
FROM analysis
WHERE analysis.completed_analysis_as_document IS NULL
Feb 27 20:19:30 0402e3 uvicorn[1161]: [generated in 0.00205s] ()
Feb 27 20:19:30 0402e3 uvicorn[1161]: COMMIT
Feb 27 20:19:30 0402e3 opentrons-api[1161]: Connecting to smoothie at port /dev/ttyAMA0
Feb 27 20:19:30 0402e3 opentrons-api-serial[1161]: smoothie: Write → b’\r\n\r\n’
Feb 27 20:19:31 0402e3 uvicorn[1161]: - “GET /networking/status HTTP/1.1” 200
Feb 27 20:19:35 0402e3 opentrons-api-serial[1161]: smoothie: Read ← b’\r\n\r\n’
Feb 27 20:19:35 0402e3 opentrons-api-serial[1161]: smoothie: retry number 0/3
Feb 27 20:19:35 0402e3 opentrons-api-serial[1161]: smoothie: Write → b’\r\n\r\n’
Feb 27 20:19:36 0402e3 uvicorn[1161]: - “GET /wifi/list HTTP/1.1” 200
Feb 27 20:19:36 0402e3 uvicorn[1161]: - “GET /networking/status HTTP/1.1” 200
Feb 27 20:19:40 0402e3 opentrons-api-serial[1161]: smoothie: Read ← b’\r\n\r\n’
Feb 27 20:19:40 0402e3 opentrons-api-serial[1161]: smoothie: retry number 1/3
Feb 27 20:19:40 0402e3 opentrons-api-serial[1161]: smoothie: Write → b’\r\n\r\n’
Feb 27 20:19:41 0402e3 uvicorn[1161]: - “GET /networking/status HTTP/1.1” 200
Feb 27 20:19:44 0402e3 uvicorn[1161]: Error response: 503 - LegacyError -
Feb 27 20:19:44 0402e3 uvicorn[1161]: - “GET /health HTTP/1.1” 503
Feb 27 20:19:46 0402e3 opentrons-api-serial[1161]: smoothie: Read ← b’\r\n\r\n’
Feb 27 20:19:46 0402e3 opentrons-api-serial[1161]: smoothie: retry number 2/3
Feb 27 20:19:46 0402e3 opentrons-api-serial[1161]: smoothie: Write → b’\r\n\r\n’
Feb 27 20:19:46 0402e3 uvicorn[1161]: - “GET /networking/status HTTP/1.1” 200
Feb 27 20:19:51 0402e3 opentrons-api-serial[1161]: smoothie: Read ← b’\r\n\r\n’
Feb 27 20:19:51 0402e3 opentrons-api-serial[1161]: smoothie: retry number 3/3
Feb 27 20:19:51 0402e3 uvicorn[1161]: - “GET /networking/status HTTP/1.1” 200
Feb 27 20:19:51 0402e3 opentrons-api-serial[1161]: smoothie: Write → b’\r\n\r\n’
Feb 27 20:19:56 0402e3 uvicorn[1161]: - “GET /networking/status HTTP/1.1” 200
Feb 27 20:19:56 0402e3 opentrons-api-serial[1161]: smoothie: Read ← b’\r\n\r\n’
Feb 27 20:19:56 0402e3 opentrons-api-serial[1161]: smoothie: retry number 0/3
Feb 27 20:19:56 0402e3 opentrons-api-serial[1161]: smoothie: Write → b’\r\n\r\n’
Feb 27 20:19:59 0402e3 uvicorn[1161]: Error response: 503 - LegacyError -
Feb 27 20:19:59 0402e3 uvicorn[1161]: - “GET /health HTTP/1.1” 503
Feb 27 20:20:01 0402e3 uvicorn[1161]: - “GET /networking/status HTTP/1.1” 200
Feb 27 20:20:01 0402e3 opentrons-api-serial[1161]: smoothie: Read ← b’\r\n\r\n’
Feb 27 20:20:01 0402e3 opentrons-api-serial[1161]: smoothie: retry number 1/3
Feb 27 20:20:02 0402e3 opentrons-api-serial[1161]: smoothie: Write → b’\r\n\r\n’
Feb 27 20:20:06 0402e3 uvicorn[1161]: - “GET /networking/status HTTP/1.1” 200
Feb 27 20:20:07 0402e3 opentrons-api-serial[1161]: smoothie: Read ← b’\r\n\r\n’
Feb 27 20:20:07 0402e3 opentrons-api-serial[1161]: smoothie: retry number 2/3
Feb 27 20:20:07 0402e3 opentrons-api-serial[1161]: smoothie: Write → b’\r\n\r\n’
Feb 27 20:20:11 0402e3 uvicorn[1161]: - “GET /networking/status HTTP/1.1” 200
Feb 27 20:20:12 0402e3 opentrons-api-serial[1161]: smoothie: Read ← b’\r\n\r\n’
Feb 27 20:20:12 0402e3 opentrons-api-serial[1161]: smoothie: retry number 3/3
Feb 27 20:20:12 0402e3 opentrons-api[1161]: [2.5K blob data]
Feb 27 20:20:12 0402e3 opentrons-api[1161]: Setting programming mode
Feb 27 20:20:13 0402e3 opentrons-api[1161]: lpc21isp -wipe -donotstart /usr/lib/firmware/smoothie-v1.1.0-25e5cea.hex /dev/ttyAMA0 115200 12000
Feb 27 20:20:13 0402e3 opentrons-api[1161]: created lpc21isp subproc in 0.02816152572631836
Feb 27 20:20:14 0402e3 uvicorn[1161]: Error response: 503 - LegacyError -
Feb 27 20:20:14 0402e3 uvicorn[1161]: - “GET /health HTTP/1.1” 503
Feb 27 20:20:16 0402e3 uvicorn[1161]: - “GET /networking/status HTTP/1.1” 200
Feb 27 20:20:21 0402e3 uvicorn[1161]: - “GET /networking/status HTTP/1.1” 200
Feb 27 20:20:26 0402e3 uvicorn[1161]: - “GET /networking/status HTTP/1.1” 200
Feb 27 20:20:30 0402e3 uvicorn[1161]: Error response: 503 - LegacyError -
Feb 27 20:20:30 0402e3 uvicorn[1161]: - “GET /health HTTP/1.1” 503
Feb 27 20:20:31 0402e3 uvicorn[1161]: - “GET /networking/status HTTP/1.1” 200
Feb 27 20:20:36 0402e3 uvicorn[1161]: - “GET /networking/status HTTP/1.1” 200
Feb 27 20:20:41 0402e3 uvicorn[1161]: - “GET /networking/status HTTP/1.1” 200
Feb 27 20:20:45 0402e3 uvicorn[1161]: Error response: 503 - LegacyError -
Feb 27 20:20:45 0402e3 uvicorn[1161]: - “GET /health HTTP/1.1” 503
Feb 27 20:20:46 0402e3 uvicorn[1161]: - “GET /networking/status HTTP/1.1” 200
Feb 27 20:20:51 0402e3 uvicorn[1161]: - “GET /networking/status HTTP/1.1” 200
Feb 27 20:20:56 0402e3 uvicorn[1161]: - “GET /networking/status HTTP/1.1” 200
Feb 27 20:21:00 0402e3 uvicorn[1161]: Error response: 503 - LegacyError -
Feb 27 20:21:00 0402e3 uvicorn[1161]: - “GET /health HTTP/1.1” 503
Feb 27 20:21:01 0402e3 uvicorn[1161]: - “GET /networking/status HTTP/1.1” 200
Feb 27 20:21:03 0402e3 opentrons-api[1161]: ran lpc21isp subproc in 50.2349579334259
Feb 27 20:21:03 0402e3 opentrons-api[1161]: Smoothie update failed: 1 b"lpc21isp version 1.97\nFile /usr/lib/firmware/smoothie-v1.1.0-25e5cea.hex:\n\tloaded…\nStart Address = 0x0003C379\n\tconverted to binary format…\n\timage size : 368852\nImage size : 368852\nSynchronizing (ESC to abort)… no answer on ‘?’\n" b’’
Feb 27 20:21:03 0402e3 opentrons-api[1161]: Exception in Thread Manager build
Traceback (most recent call last):
File “usr/lib/python3.7/site-packages/opentrons/hardware_control/thread_manager.py”, line 276, in _build_and_start_loop
File “usr/lib/python3.7/asyncio/base_events.py”, line 573, in run_until_complete
File “usr/lib/python3.7/site-packages/opentrons/hardware_control/api.py”, line 214, in build_hardware_controller
File “usr/lib/python3.7/site-packages/opentrons/hardware_control/backends/controller.py”, line 315, in update_firmware
File “usr/lib/python3.7/site-packages/opentrons/drivers/smoothie_drivers/driver_3_0.py”, line 1914, in update_firmware
RuntimeError: Failed to program smoothie: 1: b’’
Feb 27 20:21:03 0402e3 uvicorn[1161]: Exception during hardware background initialization.
Traceback (most recent call last):
File “usr/lib/python3.7/site-packages/robot_server/hardware.py”, line 505, in _initialize_hardware_api
File “usr/lib/python3.7/site-packages/robot_server/hardware.py”, line 430, in _initialize_ot2_robot
File “usr/lib/python3.7/site-packages/opentrons/init.py”, line 149, in initialize
File “usr/lib/python3.7/site-packages/opentrons/init.py”, line 131, in _create_thread_manager
File “usr/lib/python3.7/site-packages/opentrons/hardware_control/thread_manager.py”, line 264, in managed_thread_ready_async
opentrons.hardware_control.thread_manager.ThreadManagerException: Failed to create Managed Object
Feb 27 20:21:06 0402e3 uvicorn[1161]: - “GET /networking/status HTTP/1.1” 200
Feb 27 20:21:11 0402e3 uvicorn[1161]: - “GET /networking/status HTTP/1.1” 200
Feb 27 20:21:15 0402e3 uvicorn[1161]: Error response: 503 - LegacyError -
Feb 27 20:21:15 0402e3 uvicorn[1161]: - “GET /health HTTP/1.1” 503
Feb 27 20:21:16 0402e3 uvicorn[1161]: - “GET /networking/status HTTP/1.1” 200
Feb 27 20:21:21 0402e3 uvicorn[1161]: - “GET /networking/status HTTP/1.1” 200
Feb 27 20:21:26 0402e3 uvicorn[1161]: - “GET /networking/status HTTP/1.1” 200
Feb 27 20:21:30 0402e3 uvicorn[1161]: Error response: 503 - LegacyError -
Feb 27 20:21:30 0402e3 uvicorn[1161]: - “GET /health HTTP/1.1” 503
Feb 27 20:21:31 0402e3 uvicorn[1161]: - “GET /networking/status HTTP/1.1” 200
Feb 27 20:21:35 0402e3 systemd[1]: opentrons-robot-server.service: Start operation timed out. Terminating.
Feb 27 20:21:35 0402e3 uvicorn[1161]: Shutting down
Feb 27 20:21:36 0402e3 uvicorn[1161]: Waiting for connections to close. (CTRL+C to force quit)
Feb 27 20:21:36 0402e3 uvicorn[1161]: - “GET /networking/status HTTP/1.1” 200
Feb 27 20:21:36 0402e3 uvicorn[1161]: Waiting for application shutdown.
Feb 27 20:21:36 0402e3 uvicorn[1161]: Pool recreating
Feb 27 20:21:36 0402e3 uvicorn[1161]: Application shutdown complete.
Feb 27 20:21:36 0402e3 uvicorn[1161]: Finished server process [1161]
Feb 27 20:21:38 0402e3 systemd[1]: opentrons-robot-server.service: Failed with result ‘timeout’.
Feb 27 20:21:38 0402e3 systemd[1]: Failed to start Opentrons Robot HTTP Server.
Feb 27 20:21:38 0402e3 systemd[1]: opentrons-robot-server.service: Service RestartSec=100ms expired, scheduling restart.
Feb 27 20:21:38 0402e3 systemd[1]: opentrons-robot-server.service: Scheduled restart job, restart counter is at 3.
Feb 27 20:21:38 0402e3 systemd[1]: Stopped Opentrons Robot HTTP Server.
Feb 27 20:21:38 0402e3 systemd[1]: Starting Opentrons Robot HTTP Server…