CoderFunda
  • Home
  • About us
    • Contact Us
    • Disclaimer
    • Privacy Policy
    • About us
  • Home
  • Php
  • HTML
  • CSS
  • JavaScript
    • JavaScript
    • Jquery
    • JqueryUI
    • Stock
  • SQL
  • Vue.Js
  • Python
  • Wordpress
  • C++
    • C++
    • C
  • Laravel
    • Laravel
      • Overview
      • Namespaces
      • Middleware
      • Routing
      • Configuration
      • Application Structure
      • Installation
    • Overview
  • DBMS
    • DBMS
      • PL/SQL
      • SQLite
      • MongoDB
      • Cassandra
      • MySQL
      • Oracle
      • CouchDB
      • Neo4j
      • DB2
      • Quiz
    • Overview
  • Entertainment
    • TV Series Update
    • Movie Review
    • Movie Review
  • More
    • Vue. Js
    • Php Question
    • Php Interview Question
    • Laravel Interview Question
    • SQL Interview Question
    • IAS Interview Question
    • PCS Interview Question
    • Technology
    • Other

03 August, 2024

Very slow NET I/O speeds between docker containers when indexing to elasticsearch

 Programing Coderfunda     August 03, 2024     No comments   

I have spent a lot of time figuring out this but I cant find the way to fix it. I have one computer that has 48gb assigned to docker(31 to Elastic and everything else to other containers). Other computer has xeon 4410y silver and 128gb ram total and same way assigned to docker. This xeon pc is rocky linux, first pc is windows with WSL2. I have tried a lot of things but none of them work. I am trying to index db data to elasticsearch but it takes centuries on wsl2, on linux its like 1m entries in few minutes, wsl is 100k in 1 hour or more. Here are things I have tried(with ES settings, wsl configuration) that was told as fix somewhere in web, but did not work -



wsl -d docker-desktop && sysctl -w vm.max_map_count=262144



in wslconfig -



kernelCommandLine=ipv6.disable=1



es settings -
{
"index": {
"refresh_interval": "60s"
}
}

{
"index": {
"number_of_replicas": 0
}
}




sudo apt install ethtool


eth0 | grep 'tcp-segmentation-offload'


sudo ethtool -K eth0 tso off sudo ethtool -k

CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
47748c962aac INDEX 0.00% 15.43MiB / 47.05GiB 0.03% 16.5MB / 29.8MB 0B / 0B 7
d05d2eb764da DB 16.42% 267.6MiB / 47.05GiB 0.56% 2.84MB / 16.3MB 0B / 0B 11
3e013b08fd13 ES 0.12% 31.13GiB / 47.05GiB 66.16% 27MB / 195kB 0B / 0B 86



Here you can see on wsl that the NET I/O are very low numbers transfered over around 1hour. On Xeon it would be already many MB. Overall WSL pc took 18+ hours to index, xeon took around 40 minutes. The db has ~58m entries. CPU usage is very low. What is going on here? How I can fix this or what I can try?


Thanks
  • Share This:  
  •  Facebook
  •  Twitter
  •  Google+
  •  Stumble
  •  Digg
Email ThisBlogThis!Share to XShare to Facebook
Newer Post Older Post Home

0 comments:

Post a Comment

Thanks

Meta

Popular Posts

  • Crawl and Index Your Website with Laravel Site Search
      Laravel Site Search   is a package by Spatie to create a full-text search index by crawling your site. You can think of it as a private Go...
  • JqueryUI - Show
    JqueryUI - Show, JqueryUI,  This chapter will discuss the show() method, which is one of the methods used to manage jQueryUI visual effe...
  • WordPress Table
    WordPress Table WordPress table is an easy way to show the data in the table format. In the past, we had used the HTML code or table plugin ...
  • CodeIgniter - Adding JS & CSS
    Adding JavaScript and CSS (Cascading Style Sheet) file in CodeIgniter is very simple. You have to create JS and CSS folder in root directo...
  • SpinLock VB.Net Example From MSDN Possibly Produces Incorrect Behaviour
    The code below is part of an example from MSDN. It is the example on how to use SpinLock but to my eye there is a race condition in it. Why ...

Categories

  • Ajax (26)
  • Bootstrap (30)
  • DBMS (42)
  • HTML (12)
  • HTML5 (45)
  • JavaScript (10)
  • Jquery (34)
  • Jquery UI (2)
  • JqueryUI (32)
  • Laravel (1017)
  • Laravel Tutorials (23)
  • Laravel-Question (6)
  • Magento (9)
  • Magento 2 (95)
  • MariaDB (1)
  • MySql Tutorial (2)
  • PHP-Interview-Questions (3)
  • Php Question (13)
  • Python (36)
  • RDBMS (13)
  • SQL Tutorial (79)
  • Vue.js Tutorial (69)
  • Wordpress (150)
  • Wordpress Theme (3)
  • codeigniter (108)
  • oops (4)
  • php (853)

Social Media Links

  • Follow on Twitter
  • Like on Facebook
  • Subscribe on Youtube
  • Follow on Instagram

Pages

  • Home
  • Contact Us
  • Privacy Policy
  • About us

Blog Archive

  • July (4)
  • September (100)
  • August (50)
  • July (56)
  • June (46)
  • May (59)
  • April (50)
  • March (60)
  • February (42)
  • January (53)
  • December (58)
  • November (61)
  • October (39)
  • September (36)
  • August (36)
  • July (34)
  • June (34)
  • May (36)
  • April (29)
  • March (82)
  • February (1)
  • January (8)
  • December (14)
  • November (41)
  • October (13)
  • September (5)
  • August (48)
  • July (9)
  • June (6)
  • May (119)
  • April (259)
  • March (122)
  • February (368)
  • January (33)
  • October (2)
  • July (11)
  • June (29)
  • May (25)
  • April (168)
  • March (93)
  • February (60)
  • January (28)
  • December (195)
  • November (24)
  • October (40)
  • September (55)
  • August (6)
  • July (48)
  • May (2)
  • January (2)
  • July (6)
  • June (6)
  • February (17)
  • January (69)
  • December (122)
  • November (56)
  • October (92)
  • September (76)
  • August (6)

Loading...

Laravel News

Loading...

Copyright © CoderFunda | Powered by Blogger
Design by Coderfunda | Blogger Theme by Coderfunda | Distributed By Coderfunda