Comments (13)
@fballiano I found the way to mitigate this for now.
On local.xml there's a section called
<initStatements><![CDATA[SET NAMES utf8]]></initStatements>
I modified it like this for the site experiencing the issue
<initStatements><![CDATA[SET NAMES utf8;SET SESSION optimizer_search_depth = 1;]]></initStatements>
The problem is successfully mitigated for my case.
The underlying issue is still there though about the multiple joins and we need to find a permanent way to do this for the queries that actually suffer from this and not for the whole connection in my opinion.
For now though this helps
from magento-lts.
Is the version you use in testing OpenMage v20.5.0 + Magento Sample Data?
from magento-lts.
I'm seeing this on a production site using OpenMage v20.5.0 so it's not using Magento Sample Data.
I've made sure it occurs with all 3rd party extensions removed as well.
It seems to come directly from core.
The more joins are added to the same table, the more the time of execution increases.
from magento-lts.
would it be possible for you to remove some of those attributes from "available in product listing"?
from magento-lts.
Actually most of them are already set to No
from magento-lts.
there are 9 visible in the query
from magento-lts.
I checked them all and only 3 were used in product listing. I changed them to No as well
from magento-lts.
Even after changing all to No I still get this query
SELECT `operating_system_idx`.`value`, COUNT(operating_system_idx.entity_id) AS `count`
FROM `catalog_product_entity` AS `e`
INNER JOIN `catalog_category_product_index` AS `cat_index` ON cat_index.product_id=e.entity_id AND cat_index.store_id=1 AND cat_index.visibility IN(2, 4) AND cat_index.category_id = '17'
INNER JOIN `catalog_product_index_price` AS `price_index` ON price_index.entity_id = e.entity_id AND price_index.website_id = '1' AND price_index.customer_group_id = 0
INNER JOIN `catalog_product_index_eav` AS `screen_resolution_idx` ON screen_resolution_idx.entity_id = e.entity_id AND screen_resolution_idx.attribute_id = 188 AND screen_resolution_idx.store_id = 1 AND screen_resolution_idx.value = '265'
INNER JOIN `catalog_product_index_eav` AS `screen_type_idx` ON screen_type_idx.entity_id = e.entity_id AND screen_type_idx.attribute_id = 189 AND screen_type_idx.store_id = 1 AND screen_type_idx.value = '39'
INNER JOIN `catalog_product_index_eav` AS `core_count_idx` ON core_count_idx.entity_id = e.entity_id AND core_count_idx.attribute_id = 193 AND core_count_idx.store_id = 1 AND core_count_idx.value = '41'
INNER JOIN `catalog_product_index_eav` AS `ram_idx` ON ram_idx.entity_id = e.entity_id AND ram_idx.attribute_id = 196 AND ram_idx.store_id = 1 AND ram_idx.value = '54'
INNER JOIN `catalog_product_index_eav` AS `primary_camera_idx` ON primary_camera_idx.entity_id = e.entity_id AND primary_camera_idx.attribute_id = 198 AND primary_camera_idx.store_id = 1 AND primary_camera_idx.value = '309'
INNER JOIN `catalog_product_index_eav` AS `megethossim_idx` ON megethossim_idx.entity_id = e.entity_id AND megethossim_idx.attribute_id = 229 AND megethossim_idx.store_id = 1 AND megethossim_idx.value = '164'
INNER JOIN `catalog_product_index_eav` AS `nfcsosto_idx` ON nfcsosto_idx.entity_id = e.entity_id AND nfcsosto_idx.attribute_id = 258 AND nfcsosto_idx.store_id = 1 AND nfcsosto_idx.value = '377'
INNER JOIN `catalog_product_index_eav` AS `operating_system_idx` ON operating_system_idx.entity_id = e.entity_id AND operating_system_idx.attribute_id = 195 AND operating_system_idx.store_id = '1'
GROUP BY `operating_system_idx`.`value`
from magento-lts.
I'm thinking that the culprit may be here as it's one of the few areas I can see that match the query about count(attribute_code_idx)
//app/code/core/Mage/Catalog/Model/Layer/Filter/Attribute.php:115
protected function _getItemsData()
{
$attribute = $this->getAttributeModel();
$this->_requestVar = $attribute->getAttributeCode();
$key = $this->getLayer()->getStateKey() . '_' . $this->_requestVar;
$data = $this->getLayer()->getAggregator()->getCacheData($key);
if ($data === null) {
$options = $attribute->getFrontend()->getSelectOptions();
$optionsCount = $this->_getResource()->getCount($this);
$data = [];
foreach ($options as $option) {
if (is_array($option['value'])) {
continue;
}
if (Mage::helper('core/string')->strlen($option['value'])) {
// Check filter type
if ($this->_getIsFilterableAttribute($attribute) == self::OPTIONS_ONLY_WITH_RESULTS) {
if (!empty($optionsCount[$option['value']])) {
$data[] = [
'label' => $option['label'],
'value' => $option['value'],
'count' => $optionsCount[$option['value']],
];
}
} else {
$data[] = [
'label' => $option['label'],
'value' => $option['value'],
'count' => $optionsCount[$option['value']] ?? 0,
];
}
}
}
$tags = [
Mage_Eav_Model_Entity_Attribute::CACHE_TAG . ':' . $attribute->getId()
];
$tags = $this->getLayer()->getStateTags($tags);
$this->getLayer()->getAggregator()->saveCacheData($data, $key, $tags);
}
return $data;
}
from magento-lts.
Check how I replicated it.
I went in a category page with 16 filters in the sidebar's layered navigation (smartphones as products which have a lot of filters)
I progressively started clicking on filters
At around 12 of them even if it was showing only one product. I reached a query like this
SELECT `manufacturer2_idx`.`value`, COUNT(manufacturer2_idx.entity_id) AS `count`
FROM `catalog_product_entity` AS `e`
INNER JOIN `catalog_category_product_index` AS `cat_index`
ON cat_index.product_id = e.entity_id AND cat_index.store_id = 1 AND
cat_index.visibility IN (2, 4) AND cat_index.category_id = '15'
INNER JOIN `catalog_product_index_price` AS `price_index`
ON price_index.entity_id = e.entity_id AND price_index.website_id = '1' AND
price_index.customer_group_id = 0
INNER JOIN `catalog_product_index_eav` AS `screen_resolution_idx`
ON screen_resolution_idx.entity_id = e.entity_id AND screen_resolution_idx.attribute_id = 188 AND
screen_resolution_idx.store_id = 1 AND screen_resolution_idx.value = '249'
INNER JOIN `catalog_product_index_eav` AS `screen_type_idx`
ON screen_type_idx.entity_id = e.entity_id AND screen_type_idx.attribute_id = 189 AND
screen_type_idx.store_id = 1 AND screen_type_idx.value = '37'
INNER JOIN `catalog_product_index_eav` AS `core_count_idx`
ON core_count_idx.entity_id = e.entity_id AND core_count_idx.attribute_id = 193 AND
core_count_idx.store_id = 1 AND core_count_idx.value = '41'
INNER JOIN `catalog_product_index_eav` AS `operating_system_idx`
ON operating_system_idx.entity_id = e.entity_id AND operating_system_idx.attribute_id = 195 AND
operating_system_idx.store_id = 1 AND operating_system_idx.value = '48'
INNER JOIN `catalog_product_index_eav` AS `ram_idx`
ON ram_idx.entity_id = e.entity_id AND ram_idx.attribute_id = 196 AND ram_idx.store_id = 1 AND
ram_idx.value = '52'
INNER JOIN `catalog_product_index_eav` AS `memory_idx`
ON memory_idx.entity_id = e.entity_id AND memory_idx.attribute_id = 197 AND
memory_idx.store_id = 1 AND memory_idx.value = '62'
INNER JOIN `catalog_product_index_eav` AS `primary_camera_idx`
ON primary_camera_idx.entity_id = e.entity_id AND primary_camera_idx.attribute_id = 198 AND
primary_camera_idx.store_id = 1 AND primary_camera_idx.value = '309'
INNER JOIN `catalog_product_index_eav` AS `second_camera_idx`
ON second_camera_idx.entity_id = e.entity_id AND second_camera_idx.attribute_id = 200 AND
second_camera_idx.store_id = 1 AND second_camera_idx.value = '87'
INNER JOIN `catalog_product_index_eav` AS `megethossim_idx`
ON megethossim_idx.entity_id = e.entity_id AND megethossim_idx.attribute_id = 229 AND
megethossim_idx.store_id = 1 AND megethossim_idx.value = '164'
INNER JOIN `catalog_product_index_eav` AS `nfcsosto_idx`
ON nfcsosto_idx.entity_id = e.entity_id AND nfcsosto_idx.attribute_id = 258 AND
nfcsosto_idx.store_id = 1 AND nfcsosto_idx.value = '377'
INNER JOIN `catalog_product_index_eav` AS `manufacturer2_idx`
ON manufacturer2_idx.entity_id = e.entity_id AND manufacturer2_idx.attribute_id = 186 AND
manufacturer2_idx.store_id = '1'
GROUP BY `manufacturer2_idx`.`value`
This query stopped working entirely giving timeout after 1 hour.
Unfortunately on the demo I can only apply 7-8 filters
They're not enough to test this on default
However check this URL:
After disabling cache open up this url which will load slightly fast.
It has 7 filters selected
Try to select the last filter
For me it took a couple of seconds more than the previous request.
Progressively adding more should showcase the issue
from magento-lts.
ah ok so those attributes are still filterable, I don't know if there's anything possible here, the query is very complex
from magento-lts.
True it's very complex but I think the problem is still that count query. Since I configured the backend not to show the count, shouldn't this count query be totally eliminated?
It also is a significant issue because it makes filters unusable after a point and also with crawlers in the mix it can bring a whole db server down because they will drill down through all the filters
from magento-lts.
I posted this on https://dba.stackexchange.com/questions/338347/why-is-this-query-running-so-slowly/338372#338372
Where Dan Black was kind enough to take a look at this. His last suggestion was to set the
optimizer_search_depth = 1
From the default value of 62 as suggested in this KB reference https://mariadb.com/kb/en/entity-attribute-value-implementation/ for EAV implementations.
As it turns out this can also be set for the session only and it made the query run instantly.
I was wondering if we can put this in the mix here because for pretty much all collection related queries that use many joins along so many tables it will give a performance boost.
I'm still exploring this over dba.stackexchange.com if you would like to add some input there as well for this.
I am thinking that basically this is only affecting OpenMage because it's using EAV system which is considered an anti-pattern. Therefore it wouldn't make sense in hosting environments to change this globally, and it may not be good for other types of queries so would it be possible to set it in the OpenMage DB adapter class?
from magento-lts.
Related Issues (20)
- Quote product prices rounding issues when catalog prices include taxes
- Upload gif image on backend system Configuration Issue HOT 2
- support smtp servers out of the box HOT 12
- In backend, add "click to copy text" feature HOT 1
- SOAP fault due to "SOAP-ERROR: Parsing Schema: can't import schema from 'http://schemas.xmlsoap.org/soap/encoding/' " HOT 25
- SOAP API: SOAP-ERROR: Parsing Schema: unexpected <complextype> in schema HOT 4
- Shopping Cart Price rule - Free Shipping weight condition
- feature request: Event for configuration actions (save/delete) on path HOT 2
- Admin filters and sorting not working after upgrader to v20.6.0 and v20.7.0 HOT 1
- Edit attributes broken? HOT 4
- Black background of PNG images with transparent background in the Wysiwyg editor HOT 11
- Skipped Zend_Validate_Barcode_Intelligentmail from autoloading HOT 1
- Improving the file name generation logic for JS and CSS files HOT 7
- After upgrade to v20.7.0 I got some alert on Admin Login HOT 16
- Fatal error after upgrading from 20.1.1 to 20.8 when clearing cache from Admin Page HOT 2
- Error when saving product with image without description HOT 6
- Problem with Configurable Swatches after PR #3686 HOT 1
- Unable to Override GroupController in OpenMage HOT 2
- N98-magerun tests fails since #2993
- Current category cache issue in navigation block
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from magento-lts.