[Satellite 6.x] The scheduled 'InventorySync::Async::InventoryScheduledSync' tasks frequently fail with a '504 Gateway Timeout' error

Solution Verified - Updated

Environment

  • Red Hat Satellite 6.11.4.1

Issue

Scheduled InventorySync::Async::InventoryScheduledSync tasks frequently fail on the 9: InventorySync::Async::InventoryFullSync (error) step with the following error:

Error:

RestClient::GatewayTimeout
504 Gateway Timeout

Resolution

This bug is being addressed by This content is not included.Bug 2018879 - Satellite inventory sync reports 504 error.

As a workaround, run the InventorySync::Async::InventoryScheduledSync task manually.

For more KB articles/solutions related to Red Hat Satellite 6.x Satellite Tasking System Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Satellite Tasking System Issues

Diagnostic Steps

  • The following backtrace is also visible in /var/log/foreman/production.log::
2023-01-11T01:00:13 [I|bac|35a527f7] Task {label: InventorySync::Async::InventoryScheduledSync, id: 6c9f6a8b-b45e-413d-9c12-352466c8c1e3, execution_plan_id: 8bd1ca30-767e-4878-a14f-075a7b5bd60c} state changed: planning 
2023-01-11T01:00:13 [I|bac|35a527f7] Task {label: InventorySync::Async::InventoryScheduledSync, id: 6c9f6a8b-b45e-413d-9c12-352466c8c1e3, execution_plan_id: 8bd1ca30-767e-4878-a14f-075a7b5bd60c} state changed: planned 
2023-01-11T01:00:13 [I|bac|35a527f7] Task {label: InventorySync::Async::InventoryScheduledSync, id: 6c9f6a8b-b45e-413d-9c12-352466c8c1e3, execution_plan_id: 8bd1ca30-767e-4878-a14f-075a7b5bd60c} state changed: running 
2023-01-11T01:00:44 [E|bac|35a527f7] 504 Gateway Timeout (RestClient::GatewayTimeout)
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/abstract_response.rb:223:in `exception_with_response'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/abstract_response.rb:103:in `return!'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:809:in `process_result'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:725:in `block in transmit'
 35a527f7 | /opt/rh/rh-ruby27/root/usr/share/ruby/net/http.rb:933:in `start'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:715:in `transmit'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:145:in `execute'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:52:in `execute'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_rh_cloud-5.0.43/app/services/foreman_rh_cloud/cloud_request.rb:11:in `execute_cloud_request'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_rh_cloud-5.0.43/app/services/foreman_rh_cloud/cert_auth.rb:19:in `execute_cloud_request'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_rh_cloud-5.0.43/lib/inventory_sync/async/query_inventory_job.rb:51:in `query_inventory'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_rh_cloud-5.0.43/lib/inventory_sync/async/query_inventory_job.rb:33:in `block (3 levels) in run'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_rh_cloud-5.0.43/lib/inventory_sync/async/query_inventory_job.rb:32:in `loop'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_rh_cloud-5.0.43/lib/inventory_sync/async/query_inventory_job.rb:32:in `block (2 levels) in run'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/relation/delegation.rb:88:in `each'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_rh_cloud-5.0.43/lib/inventory_sync/async/query_inventory_job.rb:23:in `block in run'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/callbacks.rb:112:in `block in run_callbacks'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_rh_cloud-5.0.43/lib/inventory_sync/async/inventory_full_sync.rb:21:in `block in setup_statuses'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/connection_adapters/abstract/database_statements.rb:278:in `transaction'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/transactions.rb:212:in `transaction'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_rh_cloud-5.0.43/lib/inventory_sync/async/inventory_full_sync.rb:19:in `setup_statuses'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_rh_cloud-5.0.43/lib/inventory_sync/async/inventory_hosts_sync.rb:15:in `block in setup_facet_transaction'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in `block in transaction'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/connection_adapters/abstract/transaction.rb:280:in `block in within_new_transaction'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/connection_adapters/abstract/transaction.rb:278:in `within_new_transaction'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in `transaction'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/transactions.rb:212:in `transaction'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_rh_cloud-5.0.43/lib/inventory_sync/async/inventory_hosts_sync.rb:14:in `setup_facet_transaction'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/callbacks.rb:139:in `run_callbacks'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_rh_cloud-5.0.43/lib/inventory_sync/async/query_inventory_job.rb:22:in `run'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:582:in `block (3 levels) in execute_run'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.3/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/execution_wrapper.rb:91:in `wrap'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.3/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action/progress.rb:17:in `run'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.3/app/lib/actions/middleware/load_setting_values.rb:20:in `run'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.3/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.3/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.3/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.3/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.3/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.3/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.3/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.3/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.3/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:32:in `run'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.3/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.3/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.3/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/world.rb:31:in `execute'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:581:in `block (2 levels) in execute_run'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:580:in `catch'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:580:in `block in execute_run'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:483:in `block in with_error_handling'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:483:in `catch'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:483:in `with_error_handling'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:575:in `execute_run'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:296:in `execute'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/director.rb:69:in `execute'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors.rb:18:in `run_user_code'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
 35a527f7 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
 35a527f7 | [ sidekiq ]
 35a527f7 | [ concurrent-ruby ]
2023-01-11T01:00:44 [I|bac|35a527f7] Task {label: InventorySync::Async::InventoryScheduledSync, id: 6c9f6a8b-b45e-413d-9c12-352466c8c1e3, execution_plan_id: 8bd1ca30-767e-4878-a14f-075a7b5bd60c} state changed: stopped  result: error
SBR
Product(s)
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.