CloudPoint policy continuously fails with "Operation already in progress on asset "

Problem

CloudPoint policy continuously fails with "Operation already in progress on asset <asset_id>".

Error Message

UI:
"Operation already in progress on asset <asset_id>"

Logs:
flexsnap-mongodb: 2019-07-17T19:00:00.883+0000 I WRITE    [conn49] update flexsnap.asset.lock query: { lockId: "97c1826b-0d87-4262-8b00-068da4624aa7", _id: "<asset_id>" } planSummary: IXSCAN { _id: 1 } update: { $set: { lockId: "97c1826b-0d87-4262-8b00-068da4624aa7" } } exception: E11000 duplicate key error collection: flexsnap.asset.lock index: _id_ dup key: { : "<asset_id>" } code:11000 numYields:0 locks:{ Global: { acquireCount: { r: 1, w: 1 } }, Database: { acquireCount: { w: 1 } }, Collection: { acquireCount: { w: 1 } } } 164ms
flexsnap-mongodb: 2019-07-17T19:00:00.883+0000 I COMMAND  [conn49] command flexsnap.$cmd command: update { update: "asset.lock", ordered: true, updates: [ { q: { lockId: "97c1826b-0d87-4262-8b00-068da4624aa7", _id: "<asset_id>" }, u: { $set: { lockId: "97c1826b-0d87-4262-8b00-068da4624aa7" } }, multi: false, upsert: true } ] } numYields:0 reslen:250 locks:{ Global: { acquireCount: { r: 1, w: 1 } }, Database: { acquireCount: { w: 1 } }, Collection: { acquireCount: { w: 1 } } } protocol:op_query 164ms

flexsnap-coordinator: Jul 17 19:00:01 8e5ecd2311ee flexsnap-coordinator[1] ThreadPoolExecutor-0_2 flexsnap.coordinator: INFO - on_group_operation_complete: Task Running Policy POLICYNAME is failed. Task-id 97c1826b-0d87-4262-8b00-068da4624aa7
flexsnap-coordinator: Jul 17 19:00:01 8e5ecd2311ee flexsnap-coordinator[1] ThreadPoolExecutor-0_2 flexsnap.workflow: ERROR - Failed to run engine (Running Policy POLICYNAME): Operation already in progress on asset <asset_id>

Cause

This likely occurred because the services were stopped at a bad time causing a condition where a lock was placed on the asset in question.

# docker exec -it flexsnap-mongodb mongo --ssl --sslCAFile /cloudpoint/keys/cacert.pem --sslPEMKeyFile /cloudpoint/keys/mongodb.pem --sslAllowInvalidHostnames --eval 'db.asset.lock.find().pretty()' flexsnap
{
        "_id" : "<asset_id>",
        "lockId" : "973b26fa-18fa-440a-9d96-fcff574792e4"
}

Solution

This can be addressed by running an MongoDB query to clean up the entry.