ethexe: do not drop service work in case of failures in rpc/contracts executions/networks

[grishasobol]

Problem

One case from life: sometimes rpc cannot provide a block events due to some internal problems. Firstly we think it caused by reorgs in block-chain, but I have cautch this error without reorgs also:

Service finished work with error: server returned an error response: error code -32001: unknown block number                             

Backtrace as additional information:

    Stack backtrace:                                                                                                                                                                      
       0: anyhow::error::<impl core::convert::From<E> for anyhow::Error>::from                                                                                                            
                 at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.86/src/backtrace.rs:27:14                                                                       
       1: <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual                                             
                 at /rustc/612a33f20b9b2c27380edbc4b26a01433ed114bc/library/core/src/result.rs:1989:27                                                                                    
       2: ethexe_observer::observer::read_events_impl::{{closure}}                                                                                                                        
                 at /home/gsobol/gear/ethexe/observer/src/observer.rs:348:49                                                                                                              
       3: ethexe_observer::observer::read_block_events::{{closure}}                                                                                                                       
                 at /home/gsobol/gear/ethexe/observer/src/observer.rs:292:10                                                                                                              
       4: ethexe_observer::query::Query::get_block_events::{{closure}}                                                                                                                    
                 at /home/gsobol/gear/ethexe/observer/src/query.rs:360:76
       5: ethexe_observer::query::Query::get_committed_blocks::{{closure}}
                 at /home/gsobol/gear/ethexe/observer/src/query.rs:89:14
       6: ethexe_observer::query::Query::propagate_meta_for_block::{{closure}}
                 at /home/gsobol/gear/ethexe/observer/src/query.rs:292:70
       7: ethexe_cli::service::Service::process_one_block::{{closure}}
                 at ./src/service.rs:316:52
       8: ethexe_cli::service::Service::process_block_event::{{closure}}
                 at ./src/service.rs:371:89
       9: ethexe_cli::service::Service::process_observer_event::{{closure}}
                 at ./src/service.rs:417:81
      10: ethexe_cli::service::Service::run_inner::{{closure}}
                 at ./src/service.rs:510:23
      11: ethexe_cli::service::Service::run::{{closure}}
                 at ./src/service.rs:595:26

For reproduction: I used modified ping test, which deploy new router on Holesky and deploy ping contracts. After that in inf loop we send PING message each 1 hour.

Possible Solution

Make a refactoring which would exclude of service drop in case of many not so important errors. Maybe this can be done together with #4406